Trace number 3263102

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.35 1246.97

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_80_17_90_7-log.shuffled-as.sat03-190.cnf
MD5SUMad30956261e72142648a01bc7c025dba
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables85
Number of clauses29282
Sum of the clauses size292650
Maximum clause length10
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 534
Number of clauses of size over 529248

Solver Data 

0.00/0.06	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263102-1304095306.cnf 
0.00/0.06	c Using SATTIMEOUT of 4800
0.00/0.06	c Using SATRAM of 15500
0.00/0.06	c Size of subset node = 32
0.00/0.06	c Variables = 85
0.00/0.06	c Clauses = 29282
0.00/0.06	c Literals = 292650
0.00/0.06	c Reduced size:
0.00/0.06	c Variables = 85
0.00/0.06	c Clauses = 29282
0.00/0.06	c Literals = 292650
0.00/0.06	c a2
0.49/0.51	c a1
1.29/1.30	c Independent sub-problems = 1
1.29/1.30	c Sub-problem with 85 variables and 29282 clauses
1.29/1.30	c using permutation method 4
1.29/1.30	c count = 1
1.29/1.30	c Ordering finished 1.290 seconds
1.29/1.30	c mapped 1.300 seconds
1.29/1.30	c permuted 1.300 seconds
1.29/1.30	c about to sort
1.29/1.31	c sorted
1.29/1.37	c clauses now 29282
1.29/1.37	c init finished 1.320 seconds
1.29/1.37	c tree built 1.370 seconds
1.29/1.37	c 
1.29/1.37	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.29/1.37	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.29/1.37	c ................................ | ..........                    |                                  1.4         0.0             
1.29/1.37	c ................................ | ..........                  1 |                                  1.4         0.7             
1.29/1.37	c ................................ | ..........                  2 |                                  1.4         1.5             
1.29/1.37	c ................................ | ..........                  3 |                                  1.4         2.2             
1.29/1.37	c ................................ | ..........                  4 |                                  1.4         2.9             
1.29/1.37	c ................................ | ..........                  5 |                                  1.4         3.6             
1.29/1.38	c ................................ | ..........               1000 | 191          714                 1.4       724.6         6967
95.90/25.43	c ................................ | ..........            2490000 | 661850       3476395            96.0     25932.1     23049117
191.73/49.87	c ................................ | ..........            4374000 | 1159654      6042868           192.0     22780.1     34529648
288.02/74.50	c ................................ | ..........            6040000 | 1589212      8418219           288.0     20970.0     38917880
383.93/99.22	c ................................ | ..........            7710000 | 2028929      10681121          384.0     20076.6     44431828
479.95/124.22	c ................................ | *.........            9401000 | 2480561      12906297          480.0     19584.2     47394519
575.85/148.95	c ................................ | *.........           10981000 | 2890091      15035371          576.0     19063.9     49561885
671.68/173.79	c ................................ | *.........           12478000 | 3292989      17024082          672.0     18567.1     52797864
767.94/198.62	c ................................ | *.........           13961000 | 3695063      19035570          768.0     18177.7     53781007
863.62/223.30	c ................................ | *.........           15566000 | 4145300      21004981          864.0     18016.2     57101740
959.74/248.46	c ................................ | **........           16956000 | 4521835      22902949          960.0     17662.3     58270955
1056.01/273.60	c ................................ | **........           18542000 | 4942860      24772813         1056.0     17558.4     60858985
1151.97/298.82	c ................................ | **........           20183000 | 5386023      26661337         1152.0     17519.2     64505873
1247.70/323.68	c ................................ | **........           21712000 | 5814477      28532634         1248.0     17396.9     66090948
1343.89/348.44	c ................................ | **........           23303000 | 6242336      30434274         1344.0     17338.0     68553864
1439.85/373.55	c ................................ | ***.......           24820000 | 6651933      32255616         1440.0     17235.5     71177768
1535.63/398.49	c ................................ | ***.......           26339000 | 7074938      34073920         1536.0     17147.7     73213522
1632.00/423.81	c ................................ | ***.......           27810000 | 7487423      35947077         1632.1     17039.7     75880707
1727.78/448.86	c ................................ | ***.......           29386000 | 7905644      37815527         1728.0     17005.7     77910846
1823.78/473.66	c ................................ | ***.......           30961000 | 8324437      39642143         1824.0     16974.0     80354522
1870.19/485.75	c *............................... | ***.......           31699948 | 8532594      40533994         1870.4     16948.4     81403028
1919.72/498.77	c *............................... | ****......           32420000 | 8734480      41448474         1920.0     16885.2     82372536
2016.02/523.51	c *............................... | ****......           33721000 | 9092902      43190869         2016.1     16726.2     85228314
2111.75/548.37	c *............................... | ****......           35029000 | 9454632      44862003         2112.0     16585.5     87639552
2207.79/573.25	c *............................... | ****......           36486000 | 9858539      46618086         2208.0     16524.3     88994614
2303.77/598.18	c *............................... | ****......           37850000 | 10231342     48409773         2304.1     16427.2     91104212
2399.89/623.14	c *............................... | *****.....           39242000 | 10610063     50225945         2400.1     16350.4     93819498
2495.65/647.99	c *............................... | *****.....           40636000 | 10990189     52016120         2496.0     16280.4     95170765
2591.89/673.03	c *............................... | *****.....           42027000 | 11372589     53807861         2592.0     16213.9     97388201
2687.87/698.03	c *............................... | *****.....           43288000 | 11714652     55506484         2688.0     16104.1     99885997
2783.91/722.83	c *............................... | *****.....           44615000 | 12079552     57253564         2784.0     16025.3    101472206
2879.87/748.04	c *............................... | ******....           45882000 | 12430395     58966236         2880.0     15931.1    103369686
2975.97/773.01	c *............................... | ******....           47124000 | 12771197     60645936         2976.0     15834.5    105053835
3071.95/798.22	c *............................... | ******....           48470000 | 13141504     62361628         3072.0     15777.8    107492334
3168.01/823.34	c *............................... | ******....           49871000 | 13525180     64085430         3168.1     15741.6    109717633
3264.00/848.20	c *............................... | ******....           51199000 | 13882742     65809830         3264.0     15685.9    111733124
3359.61/873.29	c *............................... | *******...           52514000 | 14240165     67505103         3360.0     15629.2    113606575
3456.01/898.40	c *............................... | *******...           53904000 | 14616050     69272216         3456.0     15597.1    115722869
3551.75/923.18	c *............................... | *******...           55384000 | 15023068     70997844         3552.1     15592.1    117946119
3647.98/948.11	c *............................... | *******...           56813000 | 15420991     72747538         3648.0     15573.7    119558921
3744.00/973.00	c *............................... | *******...           58239000 | 15817430     74527026         3744.0     15555.2    122337895
3839.76/998.05	c *............................... | ********..           59609000 | 16196255     76220576         3840.0     15523.2    124187094
3935.80/1022.96	c *............................... | ********..           61020000 | 16583702     77794151         3936.0     15503.0    127152168
4031.67/1047.69	c *............................... | ********..           62516000 | 17002221     79489334         4032.0     15504.8    128863591
4127.74/1072.37	c *............................... | ********..           63948000 | 17399362     81241441         4128.0     15491.2    130498236
4223.67/1097.29	c *............................... | ********..           65294000 | 17767155     82948609         4224.1     15457.7    132680966
4319.67/1122.19	c *............................... | *********.           66524000 | 18104123     84593055         4320.0     15399.0    134607756
4415.90/1146.93	c *............................... | *********.           67761000 | 18441321     86232391         4416.0     15344.4    135786706
4511.81/1171.75	c *............................... | *********.           68998000 | 18779288     87890922         4512.0     15292.1    136661963
4607.94/1196.87	c *............................... | *********.           70310000 | 19145935     89530824         4608.2     15257.6    137615040
4703.75/1221.66	c *............................... | *********.           71608000 | 19509606     91104443         4704.0     15222.8    140535422
4721.86/1226.30	c **.............................. | *********.           71811527 | 19567580     91386234         4721.9     15208.2    140978290
4721.86/1226.30	c **.............................. | *********.           71811528 | 19567580     91386236         4721.9     15208.2    140978290
4799.82/1246.65	c CPU limit (4800) reached
4799.82/1246.65	c Time = 4800.02 seconds
4799.82/1246.65	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263102-1304095306/watcher-3263102-1304095306 -o /tmp/evaluation-result-3263102-1304095306/solver-3263102-1304095306 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263102-1304095306.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.73 8.09 8.07 5/181 16456
/proc/meminfo: memFree=30811388/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=8972 CPUtime=0 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) R 16454 16456 16318 0 -1 4202496 328 0 0 0 0 0 0 0 25 0 1 0 521129345 9187328 270 33554432000 4194304 4338100 140736966749840 18446744073709551615 4223897 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/16456/statm: 2243 270 147 36 0 148 0

[startup+0.0974671 s]
/proc/loadavg: 7.73 8.09 8.07 5/181 16456
/proc/meminfo: memFree=30811388/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=36612 CPUtime=0.08 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) R 16454 16456 16318 0 -1 4202496 4813 0 0 0 8 0 0 0 25 0 1 0 521129345 37490688 4579 33554432000 4194304 4338100 140736966749840 18446744073709551615 4244820 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/16456/statm: 9153 4584 348 36 0 7058 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 36612

[startup+0.100463 s]
/proc/loadavg: 7.73 8.09 8.07 5/181 16456
/proc/meminfo: memFree=30811388/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=36612 CPUtime=0.09 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) R 16454 16456 16318 0 -1 4202496 4956 0 0 0 9 0 0 0 25 0 1 0 521129345 37490688 4711 33554432000 4194304 4338100 140736966749840 18446744073709551615 4244820 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/16456/statm: 9153 4713 337 36 0 7058 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 36612

[startup+0.300429 s]
/proc/loadavg: 7.73 8.09 8.07 5/181 16456
/proc/meminfo: memFree=30811388/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=57512 CPUtime=0.29 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) R 16454 16456 16318 0 -1 4202496 6184 0 0 0 29 0 0 0 25 0 1 0 521129345 58892288 5855 33554432000 4194304 4338100 140736966749840 18446744073709551615 251813635793 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/16456/statm: 14378 5855 273 36 0 12283 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 57512

[startup+0.700362 s]
/proc/loadavg: 7.73 8.09 8.07 5/181 16456
/proc/meminfo: memFree=30811388/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=17304 CPUtime=0.69 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) R 16454 16456 16318 0 -1 4202496 6217 0 0 0 69 0 0 0 25 0 1 0 521129345 17719296 2364 33554432000 4194304 4338100 140736966749840 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/16456/statm: 4326 2364 204 36 0 2231 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 17304

[startup+1.50023 s]
/proc/loadavg: 7.73 8.09 8.07 6/182 16457
/proc/meminfo: memFree=30752512/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=212832 CPUtime=1.84 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 10140 0 0 0 183 1 0 0 25 0 5 0 521129345 217939968 6238 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 53208 6239 176 36 0 51113 0
Current children cumulated CPU time (s) 1.84
Current children cumulated vsize (KiB) 212832

[startup+3.10096 s]
/proc/loadavg: 7.73 8.09 8.07 9/186 16461
/proc/meminfo: memFree=30635920/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=323500 CPUtime=8.22 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 40651 0 0 0 813 9 0 0 25 0 5 0 521129345 331264000 36749 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 80875 36749 176 36 0 78780 0
[pid=16456/tid=16458] ppid=16454 vsize=323500 CPUtime=1.71 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 9018 0 0 0 169 2 0 0 25 0 5 0 521129483 331264000 36749 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=323500 CPUtime=1.7 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 8152 0 0 0 168 2 0 0 25 0 5 0 521129483 331264000 36750 33554432000 4194304 4338100 140736966749840 18446744073709551615 4239191 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=323500 CPUtime=1.7 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) S 16454 16456 16318 0 -1 4202560 8382 0 0 0 169 1 0 0 25 0 5 0 521129483 331264000 36750 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826052870 0 0 4096 0 18446604453985334208 0 0 -1 0 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=323500 CPUtime=1.7 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 8419 0 0 0 169 1 0 0 25 0 5 0 521129483 331264000 36750 33554432000 4194304 4338100 140736966749840 18446744073709551615 4263568 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 8.22
Current children cumulated vsize (KiB) 323500

[startup+6.30043 s]
/proc/loadavg: 7.75 8.09 8.06 9/186 16461
/proc/meminfo: memFree=30349428/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=477648 CPUtime=20.9 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 80355 0 0 0 2072 18 0 0 25 0 5 0 521129345 489111552 76453 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 119412 76453 176 36 0 117317 0
[pid=16456/tid=16458] ppid=16454 vsize=477648 CPUtime=4.87 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 19025 0 0 0 483 4 0 0 25 0 5 0 521129483 489111552 76453 33554432000 4194304 4338100 140736966749840 18446744073709551615 4288134 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=477648 CPUtime=4.88 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 15421 0 0 0 484 4 0 0 25 0 5 0 521129483 489111552 76453 33554432000 4194304 4338100 140736966749840 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=477648 CPUtime=4.89 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 19273 0 0 0 485 4 0 0 25 0 5 0 521129483 489111552 76453 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=477648 CPUtime=4.85 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 19956 0 0 0 481 4 0 0 18 0 5 0 521129483 489111552 76454 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 20.9
Current children cumulated vsize (KiB) 477648

[startup+12.7004 s]
/proc/loadavg: 7.85 8.10 8.07 9/186 16461
/proc/meminfo: memFree=29957372/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=736256 CPUtime=46.18 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 133830 0 0 0 4588 30 0 0 25 0 5 0 521129345 753926144 129928 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 184064 129928 176 36 0 181969 0
[pid=16456/tid=16458] ppid=16454 vsize=736256 CPUtime=11.16 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 34842 0 0 0 1108 8 0 0 25 0 5 0 521129483 753926144 129929 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=736256 CPUtime=11.2 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 26548 0 0 0 1114 6 0 0 18 0 5 0 521129483 753926144 129929 33554432000 4194304 4338100 140736966749840 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=736256 CPUtime=11.21 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 33424 0 0 0 1115 6 0 0 25 0 5 0 521129483 753926144 129929 33554432000 4194304 4338100 140736966749840 18446744073709551615 4288158 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=736256 CPUtime=11.19 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 32337 0 0 0 1113 6 0 0 25 0 5 0 521129483 753926144 129929 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 46.18
Current children cumulated vsize (KiB) 736256

[startup+25.5012 s]
/proc/loadavg: 7.95 8.11 8.07 9/186 16462
/proc/meminfo: memFree=29377424/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=1035224 CPUtime=96.3 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 216881 0 0 0 9575 55 0 0 25 0 5 0 521129345 1060069376 212979 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 258806 212979 176 36 0 256711 0
[pid=16456/tid=16458] ppid=16454 vsize=1035224 CPUtime=23.58 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 52801 0 0 0 2344 14 0 0 25 0 5 0 521129483 1060069376 212979 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=1035224 CPUtime=23.71 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 54375 0 0 0 2357 14 0 0 25 0 5 0 521129483 1060069376 212979 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=1035224 CPUtime=23.81 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 54658 0 0 0 2368 13 0 0 25 0 5 0 521129483 1060069376 212979 33554432000 4194304 4338100 140736966749840 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=1035224 CPUtime=23.79 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 48367 0 0 0 2368 11 0 0 25 0 5 0 521129483 1060069376 212979 33554432000 4194304 4338100 140736966749840 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 96.3
Current children cumulated vsize (KiB) 1035224

[startup+51.1009 s]
/proc/loadavg: 7.89 8.09 8.07 9/186 16463
/proc/meminfo: memFree=28609288/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=1465080 CPUtime=196.91 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 336582 0 0 0 19596 95 0 0 25 0 5 0 521129345 1500241920 332680 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 366270 332680 176 36 0 364175 0
[pid=16456/tid=16458] ppid=16454 vsize=1465080 CPUtime=48.54 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 91031 0 0 0 4828 26 0 0 25 0 5 0 521129483 1500241920 332681 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=1465080 CPUtime=48.88 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 80600 0 0 0 4865 23 0 0 25 0 5 0 521129483 1500241920 332681 33554432000 4194304 4338100 140736966749840 18446744073709551615 251813901345 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=1465080 CPUtime=49.19 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 80067 0 0 0 4898 21 0 0 25 0 5 0 521129483 1500241920 332681 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=1465080 CPUtime=48.88 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 78205 0 0 0 4866 22 0 0 25 0 5 0 521129483 1500241920 332681 33554432000 4194304 4338100 140736966749840 18446744073709551615 4279490 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 196.91
Current children cumulated vsize (KiB) 1465080

[startup+102.306 s]
/proc/loadavg: 7.94 8.07 8.06 9/186 16464
/proc/meminfo: memFree=27816404/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=2018196 CPUtime=396.08 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 447086 0 0 0 39461 147 0 0 25 0 5 0 521129345 2066632704 443184 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 504549 443184 176 36 0 502454 0
[pid=16456/tid=16458] ppid=16454 vsize=2018196 CPUtime=97.79 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 122533 0 0 0 9735 44 0 0 25 0 5 0 521129483 2066632704 443184 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=2018196 CPUtime=98.65 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 106526 0 0 0 9831 34 0 0 25 0 5 0 521129483 2066632704 443184 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=2018196 CPUtime=99.41 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 110231 0 0 0 9907 34 0 0 25 0 5 0 521129483 2066632704 443184 33554432000 4194304 4338100 140736966749840 18446744073709551615 4263517 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=2018196 CPUtime=98.84 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 101116 0 0 0 9850 34 0 0 25 0 5 0 521129483 2066632704 443184 33554432000 4194304 4338100 140736966749840 18446744073709551615 251813635297 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 396.08
Current children cumulated vsize (KiB) 2018196

[startup+162.3 s]
/proc/loadavg: 7.95 8.04 8.05 9/186 16465
/proc/meminfo: memFree=27087008/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=2287288 CPUtime=627.97 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 545438 0 0 0 62593 204 0 0 25 0 5 0 521129345 2342182912 541536 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 571822 541536 176 36 0 569727 0
[pid=16456/tid=16458] ppid=16454 vsize=2287288 CPUtime=154.68 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 182151 0 0 0 15399 69 0 0 25 0 5 0 521129483 2342182912 541536 33554432000 4194304 4338100 140736966749840 18446744073709551615 4286144 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=2287288 CPUtime=156.86 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 121948 0 0 0 15638 48 0 0 25 0 5 0 521129483 2342182912 541536 33554432000 4194304 4338100 140736966749840 18446744073709551615 4284096 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=2287288 CPUtime=157.74 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 121431 0 0 0 15735 39 0 0 25 0 5 0 521129483 2342182912 541536 33554432000 4194304 4338100 140736966749840 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=2287288 CPUtime=157.3 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 113228 0 0 0 15684 46 0 0 25 0 5 0 521129483 2342182912 541536 33554432000 4194304 4338100 140736966749840 18446744073709551615 4288196 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 627.97
Current children cumulated vsize (KiB) 2287288

[startup+222.301 s]
/proc/loadavg: 7.80 7.98 8.03 9/186 16467
/proc/meminfo: memFree=26483840/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=2699412 CPUtime=860.05 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 622103 0 0 0 85753 252 0 0 25 0 5 0 521129345 2764197888 618201 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 674853 618201 176 36 0 672758 0
[pid=16456/tid=16458] ppid=16454 vsize=2699412 CPUtime=211.81 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 230467 0 0 0 21089 92 0 0 25 0 5 0 521129483 2764197888 618201 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0

################
# More data... #
################

/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 722297 0 0 0 77436 279 0 0 25 0 5 0 521129483 6140616704 1420260 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=5996696 CPUtime=794.71 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 233767 0 0 0 79311 160 0 0 25 0 5 0 521129483 6140616704 1420260 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=5996696 CPUtime=796.35 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 241589 0 0 0 79489 146 0 0 25 0 5 0 521129483 6140616704 1420260 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=5996696 CPUtime=794.49 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 219829 0 0 0 79302 147 0 0 25 0 5 0 521129483 6140616704 1420260 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3164.11
Current children cumulated vsize (KiB) 5996696

[startup+882.3 s]
/proc/loadavg: 7.66 7.71 7.83 9/186 16485
/proc/meminfo: memFree=20573440/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=6401640 CPUtime=3394.64 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 1505302 0 0 0 338683 781 0 0 25 0 5 0 521129345 6555279360 1501400 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 1600410 1501400 176 36 0 1598315 0
[pid=16456/tid=16458] ppid=16454 vsize=6401640 CPUtime=833.76 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 766360 0 0 0 83082 294 0 0 25 0 5 0 521129483 6555279360 1501400 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=6401640 CPUtime=852.56 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 241824 0 0 0 85085 171 0 0 25 0 5 0 521129483 6555279360 1501400 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=6401640 CPUtime=854.24 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 255575 0 0 0 85267 157 0 0 25 0 5 0 521129483 6555279360 1501400 33554432000 4194304 4338100 140736966749840 18446744073709551615 4263564 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=6401640 CPUtime=852.67 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 234863 0 0 0 85110 157 0 0 25 0 5 0 521129483 6555279360 1501400 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3394.64
Current children cumulated vsize (KiB) 6401640

[startup+942.3 s]
/proc/loadavg: 7.88 7.76 7.84 9/186 16517
/proc/meminfo: memFree=23965908/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=6728632 CPUtime=3625.65 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 1587407 0 0 0 361738 827 0 0 25 0 5 0 521129345 6890119168 1583505 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 1682158 1583505 176 36 0 1680063 0
[pid=16456/tid=16458] ppid=16454 vsize=6728632 CPUtime=890.47 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 814801 0 0 0 88736 311 0 0 23 0 5 0 521129483 6890119168 1583505 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=6728632 CPUtime=910.74 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 253894 0 0 0 90894 180 0 0 25 0 5 0 521129483 6890119168 1583505 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=6728632 CPUtime=912.27 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 269377 0 0 0 91061 166 0 0 25 0 5 0 521129483 6890119168 1583505 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=6728632 CPUtime=910.77 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 242655 0 0 0 90909 168 0 0 25 0 5 0 521129483 6890119168 1583505 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3625.65
Current children cumulated vsize (KiB) 6728632

[startup+1002.3 s]
/proc/loadavg: 7.72 7.74 7.82 9/186 16519
/proc/meminfo: memFree=23226568/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=7007272 CPUtime=3856.41 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 1668611 0 0 0 384765 876 0 0 25 0 5 0 521129345 7175446528 1664709 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 1751818 1664709 176 36 0 1749723 0
[pid=16456/tid=16458] ppid=16454 vsize=7007272 CPUtime=947.05 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 868078 0 0 0 94375 330 0 0 25 0 5 0 521129483 7175446528 1664709 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=7007272 CPUtime=969.11 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 261827 0 0 0 96722 189 0 0 21 0 5 0 521129483 7175446528 1664709 33554432000 4194304 4338100 140736966749840 18446744073709551615 4263536 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=7007272 CPUtime=970.12 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 279120 0 0 0 96836 176 0 0 25 0 5 0 521129483 7175446528 1664709 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=7007272 CPUtime=968.71 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 252906 0 0 0 96693 178 0 0 25 0 5 0 521129483 7175446528 1664709 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3856.41
Current children cumulated vsize (KiB) 7007272

[startup+1062.3 s]
/proc/loadavg: 7.94 7.80 7.83 9/186 16520
/proc/meminfo: memFree=22673256/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=7423052 CPUtime=4089 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 1749501 0 0 0 407979 921 0 0 25 0 5 0 521129345 7601205248 1745599 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 1855763 1745599 176 36 0 1853668 0
[pid=16456/tid=16458] ppid=16454 vsize=7423052 CPUtime=1004.31 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 914784 0 0 0 100085 346 0 0 25 0 5 0 521129483 7601205248 1745599 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=7423052 CPUtime=1027.55 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 271169 0 0 0 102557 198 0 0 25 0 5 0 521129483 7601205248 1745599 33554432000 4194304 4338100 140736966749840 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=7423052 CPUtime=1028.39 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 287238 0 0 0 102653 186 0 0 25 0 5 0 521129483 7601205248 1745599 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=7423052 CPUtime=1027.33 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 269630 0 0 0 102545 188 0 0 25 0 5 0 521129483 7601205248 1745599 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4089
Current children cumulated vsize (KiB) 7423052

[startup+1122.3 s]
/proc/loadavg: 7.72 7.76 7.81 9/186 16522
/proc/meminfo: memFree=22018740/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=7728636 CPUtime=4320.46 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 1817542 0 0 0 431075 971 0 0 25 0 5 0 521129345 7914123264 1813640 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 1932159 1813641 176 36 0 1930064 0
[pid=16456/tid=16458] ppid=16454 vsize=7728636 CPUtime=1061.42 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 958395 0 0 0 105777 365 0 0 25 0 5 0 521129483 7914123264 1813641 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=7728636 CPUtime=1086.07 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 278743 0 0 0 108400 207 0 0 25 0 5 0 521129483 7914123264 1813641 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=7728636 CPUtime=1086.31 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 294428 0 0 0 108434 197 0 0 25 0 5 0 521129483 7914123264 1813641 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=7728636 CPUtime=1085.25 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 279297 0 0 0 108325 200 0 0 25 0 5 0 521129483 7914123264 1813641 33554432000 4194304 4338100 140736966749840 18446744073709551615 251813901345 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4320.46
Current children cumulated vsize (KiB) 7728636

[startup+1182.3 s]
/proc/loadavg: 7.76 7.75 7.81 9/186 16533
/proc/meminfo: memFree=21638088/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=7885376 CPUtime=4552.22 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 1867287 0 0 0 454207 1015 0 0 25 0 5 0 521129345 8074625024 1863385 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 1971344 1863385 176 36 0 1969249 0
[pid=16456/tid=16458] ppid=16454 vsize=7885376 CPUtime=1118.65 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 981617 0 0 0 111487 378 0 0 25 0 5 0 521129483 8074625024 1863385 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=7885376 CPUtime=1144.16 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 287153 0 0 0 114199 217 0 0 25 0 5 0 521129483 8074625024 1863385 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=7885376 CPUtime=1144.56 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 303441 0 0 0 114249 207 0 0 25 0 5 0 521129483 8074625024 1863385 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=7885376 CPUtime=1143.44 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 288396 0 0 0 114134 210 0 0 25 0 5 0 521129483 8074625024 1863385 33554432000 4194304 4338100 140736966749840 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4552.22
Current children cumulated vsize (KiB) 7885376

[startup+1242.3 s]
/proc/loadavg: 7.97 7.81 7.82 9/186 16534
/proc/meminfo: memFree=21007496/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=8200832 CPUtime=4782.87 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) S 16454 16456 16318 0 -1 4202496 1939958 0 0 0 477228 1059 0 0 25 0 5 0 521129345 8397651968 1936056 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826042853 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 2050208 1936056 176 36 0 2048113 0
[pid=16456/tid=16458] ppid=16454 vsize=8200832 CPUtime=1175.5 cores=0,2,4,6
/proc/16456/task/16458/stat : 16458 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 1027658 0 0 0 117158 392 0 0 25 0 5 0 521129483 8397651968 1936056 33554432000 4194304 4338100 140736966749840 18446744073709551615 251813635160 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=16456/tid=16459] ppid=16454 vsize=8200832 CPUtime=1201.94 cores=0,2,4,6
/proc/16456/task/16459/stat : 16459 (ttsth-5-4) R 16454 16456 16318 0 -1 4202560 297030 0 0 0 119966 228 0 0 25 0 5 0 521129483 8397651968 1936056 33554432000 4194304 4338100 140736966749840 18446744073709551615 4234209 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=16456/tid=16460] ppid=16454 vsize=8200832 CPUtime=1202.71 cores=0,2,4,6
/proc/16456/task/16460/stat : 16460 (ttsth-5-4) S 16454 16456 16318 0 -1 4202560 312651 0 0 0 120053 218 0 0 25 0 5 0 521129483 8397651968 1936056 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826052870 0 0 4096 0 18446604453985334208 0 0 -1 0 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=8200832 CPUtime=1201.31 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) S 16454 16456 16318 0 -1 4202560 295939 0 0 0 119913 218 0 0 18 0 5 0 521129483 8397651968 1936056 33554432000 4194304 4338100 140736966749840 18446744073709551615 251826052870 0 0 4096 0 18446604453985334208 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4782.87
Current children cumulated vsize (KiB) 8200832



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1246.7 s]
/proc/loadavg: 8.05 7.83 7.83 6/183 16535
/proc/meminfo: memFree=22082824/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=0 CPUtime=4800.07 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) Z 16454 16456 16318 0 -1 4203548 1942134 0 0 0 478940 1067 0 0 18 0 2 0 521129345 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 0 0 0 0 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=0 CPUtime=1205.64 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4203588 296510 0 0 0 120340 224 0 0 25 0 2 0 521129483 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.07
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1246.81 s]
/proc/loadavg: 8.05 7.83 7.83 6/183 16535
/proc/meminfo: memFree=22082824/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=0 CPUtime=4800.18 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) Z 16454 16456 16318 0 -1 4203548 1942134 0 0 0 478940 1078 0 0 18 0 2 0 521129345 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 0 0 0 0 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=0 CPUtime=1205.75 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4203588 296510 0 0 0 120340 235 0 0 25 0 2 0 521129483 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 0

[startup+1246.9 s]
/proc/loadavg: 8.05 7.83 7.83 6/183 16535
/proc/meminfo: memFree=22082824/32950928 swapFree=67111448/67111528
[pid=16456] ppid=16454 vsize=0 CPUtime=4800.27 cores=0,2,4,6
/proc/16456/stat : 16456 (ttsth-5-4) Z 16454 16456 16318 0 -1 4203548 1942134 0 0 0 478940 1087 0 0 18 0 2 0 521129345 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16456/statm: 0 0 0 0 0 0 0
[pid=16456/tid=16461] ppid=16454 vsize=0 CPUtime=1205.84 cores=0,2,4,6
/proc/16456/task/16461/stat : 16461 (ttsth-5-4) R 16454 16456 16318 0 -1 4203588 296510 0 0 0 120340 244 0 0 25 0 2 0 521129483 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.27
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1246.97
CPU time (s): 4800.35
CPU user time (s): 4789.41
CPU system time (s): 10.9393
CPU usage (%): 384.961
Max. virtual memory (cumulated for all children) (KiB): 8202548

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4789.41
system time used= 10.9393
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1942134
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 181527
involuntary context switches= 25063

runsolver used 2.01969 second user time and 4.6163 second system time

The end

Launcher Data 

Begin job on node126 at 2011-04-29 18:41:46
IDJOB=3263102
IDBENCH=83924
IDSOLVER=1654
FILE ID=node126/3263102-1304095306
RUNJOBID= node126-1304094101-16336
PBS_JOBID= 13170590
Free space on /tmp= 71596 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_80_17_90_7-log.shuffled-as.sat03-190.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263102-1304095306/watcher-3263102-1304095306 -o /tmp/evaluation-result-3263102-1304095306/solver-3263102-1304095306 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263102-1304095306.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= ad30956261e72142648a01bc7c025dba
RANDOM SEED=1343102158

node126.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.783
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30811668 kB
Buffers:        246120 kB
Cached:        1399212 kB
SwapCached:         76 kB
Active:         234540 kB
Inactive:      1593544 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30811668 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:            2252 kB
Writeback:           0 kB
AnonPages:      182848 kB
Mapped:          15248 kB
Slab:           246972 kB
PageTables:       4760 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   377196 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71596 MiB
End job on node126 at 2011-04-29 19:02:35