Trace number 3274156

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.45 1204.63

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n240-s78945233-sat.cnf
MD5SUM1f90ac4288669aff970e47f4201b5ea4
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark10.2234
Satisfiable
(Un)Satisfiability was proved
Number of variables240
Number of clauses576
Sum of the clauses size1440
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2480
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 596
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3274156-1304060210.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 240
0.00/0.00	c Clauses = 576
0.00/0.00	c Literals = 1440
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 240
0.00/0.00	c Clauses = 576
0.00/0.00	c Literals = 1440
0.00/0.00	c a2
0.00/0.00	c a1
3.10/3.19	c Independent sub-problems = 1
3.10/3.19	c Sub-problem with 240 variables and 576 clauses
3.10/3.19	c using permutation method 3
3.10/3.19	c Ordering finished 3.190 seconds
3.10/3.19	c mapped 3.190 seconds
3.10/3.19	c permuted 3.190 seconds
3.10/3.19	c about to sort
3.10/3.19	c sorted
3.10/3.20	c clauses now 576
3.10/3.20	c init finished 3.200 seconds
3.10/3.20	c tree built 3.200 seconds
3.10/3.20	c 
3.10/3.20	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
3.10/3.20	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
3.10/3.20	c ................................ | ..........                    |                                  3.2         0.0             
3.10/3.20	c ................................ | ..........                  1 |                                  3.2         0.3             
3.10/3.20	c ................................ | ..........                  2 |                                  3.2         0.6             
3.10/3.20	c ................................ | ..........                  3 |                                  3.2         0.9             
3.10/3.20	c ................................ | ..........                  4 |                                  3.2         1.2             
3.10/3.20	c ................................ | ..........                  5 |                                  3.2         1.6             
3.20/3.21	c ................................ | ..........               1000 | 95           1189                3.2       310.6        19095
96.22/26.51	c ................................ | ..........             565000 | 103299       232093             96.3      5868.3     10380890
192.10/50.60	c ................................ | ..........             773000 | 139862       337794            192.5      4016.0     14945233
288.33/74.60	c ................................ | ..........            1047000 | 186854       442498            288.4      3631.0     19070725
384.16/98.67	c ................................ | ..........            1208000 | 216826       506617            384.5      3142.1     22066161
480.42/122.74	c ................................ | *.........            1390000 | 250464       560483            480.6      2892.3     24975940
576.28/146.78	c ................................ | *.........            1494000 | 267260       613603            576.6      2591.1     26765239
672.90/170.96	c ................................ | *.........            1630000 | 291381       668710            673.1      2421.5     28377073
768.37/194.89	c ................................ | *.........            1718000 | 309004       702616            768.8      2234.8     30045068
864.26/218.82	c ................................ | *.........            1821000 | 328350       739875            864.3      2106.8     31970106
960.11/242.83	c ................................ | **........            1910000 | 345920       782983            960.3      1989.0     34151686
1056.21/266.93	c ................................ | **........            2205000 | 403201       884247           1056.3      2087.4     40091087
1152.35/291.05	c ................................ | **........            2385000 | 436271       946550           1152.6      2069.3     41563044
1249.42/315.38	c ................................ | **........            2492000 | 459115       974237           1249.7      1994.0     43562122
1344.88/339.29	c ................................ | **........            2559000 | 474120       991342           1345.2      1902.2     44656826
1440.75/363.22	c ................................ | ***.......            2650000 | 494545       1011300          1440.8      1839.2     46692902
1536.57/387.25	c ................................ | ***.......            2727000 | 512529       1034023          1536.8      1774.5     47470564
1632.81/411.36	c ................................ | ***.......            2796000 | 527227       1054281          1633.0      1712.2     47509494
1729.06/435.40	c ................................ | ***.......            2884000 | 545851       1075378          1729.1      1667.9     48070714
1824.53/459.34	c ................................ | ***.......            2945000 | 559391       1094461          1824.7      1614.0     48840596
1919.94/483.25	c ................................ | ****......            3034000 | 575978       1127275          1920.2      1580.1     49787357
2015.67/507.29	c ................................ | ****......            3229000 | 605211       1193570          2016.0      1601.7     50551602
2112.22/531.41	c ................................ | ****......            3362000 | 624320       1243595          2112.3      1591.7     50890578
2208.86/555.65	c ................................ | ****......            3482000 | 640664       1287440          2209.1      1576.2     52007148
2304.26/579.51	c ................................ | ****......            3638000 | 666004       1355928          2304.3      1578.8     54326407
2400.41/603.65	c ................................ | *****.....            3795000 | 696430       1404586          2400.6      1580.8     56421386
2496.15/627.60	c ................................ | *****.....            3939000 | 724665       1440537          2496.2      1578.0     57954391
2593.21/651.97	c ................................ | *****.....            4054000 | 746248       1487628          2593.5      1563.1     58806551
2688.21/675.73	c ................................ | *****.....            4151000 | 768055       1528716          2688.3      1544.1     59133705
2784.47/699.87	c ................................ | *****.....            4292000 | 791462       1590341          2784.8      1541.3     61205350
2880.74/723.99	c ................................ | ******....            4377000 | 810556       1603245          2881.1      1519.2     63159496
2976.22/747.86	c ................................ | ******....            4462000 | 827758       1626954          2976.5      1499.1     65146998
3072.77/772.02	c ................................ | ******....            4585000 | 849232       1665312          3072.9      1492.1     65751246
3168.24/795.92	c ................................ | ******....            4680000 | 863988       1708886          3168.3      1477.1     66612447
3264.07/819.94	c ................................ | ******....            4736000 | 874664       1741010          3264.2      1450.9     67118567
3361.15/844.29	c ................................ | *******...            4824000 | 890864       1770147          3361.5      1435.1     68691172
3457.39/868.30	c ................................ | *******...            4886000 | 902012       1792368          3457.4      1413.2     69667789
3552.48/892.19	c ................................ | *******...            4955000 | 914952       1820864          3552.8      1394.7     70951485
3649.52/916.41	c ................................ | *******...            5023000 | 928296       1849457          3649.6      1376.3     71994304
3744.16/940.17	c ................................ | *******...            5093000 | 941984       1869032          3744.4      1360.1     72120521
3841.24/964.42	c ................................ | ********..            5132000 | 950912       1879731          3841.3      1336.0     72368737
3935.85/988.19	c ................................ | ********..            5189000 | 963546       1892095          3936.2      1318.3     72998084
4032.12/1012.28	c ................................ | ********..            5230000 | 972630       1911969          4032.4      1297.0     73930266
4128.39/1036.32	c ................................ | ********..            5281000 | 984014       1925959          4128.5      1279.2     75042117
4225.06/1060.57	c ................................ | ********..            5340000 | 995865       1947740          4225.4      1263.8     76255802
4320.96/1084.57	c ................................ | *********.            5374000 | 1003493      1958090          4321.2      1243.6     77207323
4417.20/1108.64	c ................................ | *********.            5413000 | 1011918      1969241          4417.4      1225.4     77353662
4512.27/1132.50	c ................................ | *********.            5469000 | 1023396      1986488          4512.7      1211.9     77627206
4607.74/1156.37	c ................................ | *********.            5506000 | 1031172      2002763          4608.0      1194.9     78273265
4704.40/1180.58	c ................................ | *********.            5549000 | 1039962      2020921          4704.7      1179.4     78695608

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3274156-1304060210/watcher-3274156-1304060210 -o /tmp/evaluation-result-3274156-1304060210/solver-3274156-1304060210 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3274156-1304060210.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.39 7.90 7.96 1/172 29827
/proc/meminfo: memFree=26593808/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=8840 CPUtime=0 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) R 29825 29827 29500 0 -1 4202496 313 0 0 0 0 0 0 0 22 0 1 0 498436490 9052160 255 33554432000 4194304 4338100 140733326829008 18446744073709551615 4246488 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/29827/statm: 2210 255 163 36 0 115 0

[startup+0.0350149 s]
/proc/loadavg: 7.39 7.90 7.96 1/172 29827
/proc/meminfo: memFree=26593808/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=8840 CPUtime=0.03 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) R 29825 29827 29500 0 -1 4202496 313 0 0 0 3 0 0 0 22 0 1 0 498436490 9052160 255 33554432000 4194304 4338100 140733326829008 18446744073709551615 228094852722 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/29827/statm: 2210 255 163 36 0 115 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 8840

[startup+0.101002 s]
/proc/loadavg: 7.39 7.90 7.96 1/172 29827
/proc/meminfo: memFree=26593808/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=8840 CPUtime=0.1 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) R 29825 29827 29500 0 -1 4202496 324 0 0 0 10 0 0 0 22 0 1 0 498436490 9052160 266 33554432000 4194304 4338100 140733326829008 18446744073709551615 228095116788 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/29827/statm: 2210 266 173 36 0 115 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 8840

[startup+0.300972 s]
/proc/loadavg: 7.39 7.90 7.96 1/172 29827
/proc/meminfo: memFree=26593808/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=8840 CPUtime=0.3 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) R 29825 29827 29500 0 -1 4202496 324 0 0 0 30 0 0 0 23 0 1 0 498436490 9052160 266 33554432000 4194304 4338100 140733326829008 18446744073709551615 4245907 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/29827/statm: 2210 266 173 36 0 115 0
Current children cumulated CPU time (s) 0.3
Current children cumulated vsize (KiB) 8840

[startup+0.700902 s]
/proc/loadavg: 7.39 7.90 7.96 1/172 29827
/proc/meminfo: memFree=26593808/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=8840 CPUtime=0.7 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) R 29825 29827 29500 0 -1 4202496 324 0 0 0 70 0 0 0 25 0 1 0 498436490 9052160 266 33554432000 4194304 4338100 140733326829008 18446744073709551615 4290761 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/29827/statm: 2210 266 173 36 0 115 0
Current children cumulated CPU time (s) 0.7
Current children cumulated vsize (KiB) 8840

[startup+1.50074 s]
/proc/loadavg: 7.39 7.90 7.96 3/178 29836
/proc/meminfo: memFree=26592420/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=8840 CPUtime=1.5 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) R 29825 29827 29500 0 -1 4202496 324 0 0 0 150 0 0 0 25 0 1 0 498436490 9052160 266 33554432000 4194304 4338100 140733326829008 18446744073709551615 4290766 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/29827/statm: 2210 266 173 36 0 115 0
Current children cumulated CPU time (s) 1.5
Current children cumulated vsize (KiB) 8840

[startup+3.10045 s]
/proc/loadavg: 7.39 7.90 7.96 6/182 29840
/proc/meminfo: memFree=26541852/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=8840 CPUtime=3.1 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) R 29825 29827 29500 0 -1 4202496 326 0 0 0 310 0 0 0 25 0 1 0 498436490 9052160 268 33554432000 4194304 4338100 140733326829008 18446744073709551615 4307599 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/29827/statm: 2210 268 175 36 0 115 0
Current children cumulated CPU time (s) 3.1
Current children cumulated vsize (KiB) 8840

[startup+6.30083 s]
/proc/loadavg: 7.19 7.85 7.94 9/186 29844
/proc/meminfo: memFree=26396844/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=350720 CPUtime=15.56 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 24913 0 0 0 1548 8 0 0 25 0 5 0 498436490 359137280 24855 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 87680 24855 196 36 0 85585 0
[pid=29827/tid=29841] ppid=29825 vsize=350720 CPUtime=3.07 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 3579 0 0 0 304 3 0 0 25 0 5 0 498436810 359137280 24855 33554432000 4194304 4338100 140733326829008 18446744073709551615 228094846102 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=350720 CPUtime=3.09 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 7307 0 0 0 308 1 0 0 25 0 5 0 498436810 359137280 24855 33554432000 4194304 4338100 140733326829008 18446744073709551615 228094846377 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=350720 CPUtime=3.09 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 5439 0 0 0 307 2 0 0 25 0 5 0 498436810 359137280 24855 33554432000 4194304 4338100 140733326829008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=350720 CPUtime=3.09 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 7579 0 0 0 308 1 0 0 25 0 5 0 498436810 359137280 24855 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 15.56
Current children cumulated vsize (KiB) 350720

[startup+12.7006 s]
/proc/loadavg: 7.34 7.87 7.95 9/186 29844
/proc/meminfo: memFree=26194628/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=460712 CPUtime=41.12 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 52827 0 0 0 4093 19 0 0 25 0 5 0 498436490 471769088 52769 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 115178 52769 196 36 0 113083 0
[pid=29827/tid=29841] ppid=29825 vsize=460712 CPUtime=9.43 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 7753 0 0 0 937 6 0 0 25 0 5 0 498436810 471769088 52769 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=460712 CPUtime=9.48 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 14523 0 0 0 946 2 0 0 25 0 5 0 498436810 471769088 52769 33554432000 4194304 4338100 140733326829008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=460712 CPUtime=9.48 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 10386 0 0 0 942 6 0 0 25 0 5 0 498436810 471769088 52769 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=460712 CPUtime=9.49 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 19156 0 0 0 946 3 0 0 25 0 5 0 498436810 471769088 52769 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 41.12
Current children cumulated vsize (KiB) 460712

[startup+25.5003 s]
/proc/loadavg: 7.56 7.90 7.95 9/186 29844
/proc/meminfo: memFree=25922716/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=580532 CPUtime=92.22 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 82714 0 0 0 9184 38 0 0 25 0 5 0 498436490 594464768 82648 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 145133 82648 198 36 0 143038 0
[pid=29827/tid=29841] ppid=29825 vsize=580532 CPUtime=22.16 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 13378 0 0 0 2202 14 0 0 25 0 5 0 498436810 594464768 82648 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=580532 CPUtime=22.28 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 23669 0 0 0 2223 5 0 0 25 0 5 0 498436810 594464768 82648 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=580532 CPUtime=22.27 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 16230 0 0 0 2216 11 0 0 25 0 5 0 498436810 594464768 82649 33554432000 4194304 4338100 140733326829008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=580532 CPUtime=22.29 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 28429 0 0 0 2222 7 0 0 25 0 5 0 498436810 594464768 82649 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 92.22
Current children cumulated vsize (KiB) 580532

[startup+51.1006 s]
/proc/loadavg: 7.71 7.90 7.95 9/186 29845
/proc/meminfo: memFree=25554660/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=731444 CPUtime=194.48 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 120920 0 0 0 19387 61 0 0 25 0 5 0 498436490 748998656 120854 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 182861 120855 198 36 0 180766 0
[pid=29827/tid=29841] ppid=29825 vsize=731444 CPUtime=47.61 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 19131 0 0 0 4739 22 0 0 25 0 5 0 498436810 748998656 120855 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=731444 CPUtime=47.88 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 32266 0 0 0 4781 7 0 0 25 0 5 0 498436810 748998656 120855 33554432000 4194304 4338100 140733326829008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=731444 CPUtime=47.87 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 22280 0 0 0 4768 19 0 0 25 0 5 0 498436810 748998656 120855 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=731444 CPUtime=47.89 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 46235 0 0 0 4777 12 0 0 25 0 5 0 498436810 748998656 120855 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 194.48
Current children cumulated vsize (KiB) 731444

[startup+102.306 s]
/proc/loadavg: 7.92 7.93 7.96 9/186 29846
/proc/meminfo: memFree=25184492/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=977596 CPUtime=398.96 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 182365 0 0 0 39791 105 0 0 25 0 5 0 498436490 1001058304 182291 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 244399 182291 198 36 0 242304 0
[pid=29827/tid=29841] ppid=29825 vsize=977596 CPUtime=98.61 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 29249 0 0 0 9824 37 0 0 25 0 5 0 498436810 1001058304 182291 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=977596 CPUtime=98.96 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 50747 0 0 0 9885 11 0 0 25 0 5 0 498436810 1001058304 182291 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=977596 CPUtime=99.07 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 32508 0 0 0 9871 36 0 0 25 0 5 0 498436810 1001058304 182291 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=977596 CPUtime=99.08 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 68852 0 0 0 9889 19 0 0 25 0 5 0 498436810 1001058304 182291 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 398.96
Current children cumulated vsize (KiB) 977596

[startup+162.301 s]
/proc/loadavg: 7.97 7.94 7.96 9/186 29848
/proc/meminfo: memFree=24730488/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=1141020 CPUtime=638.58 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 224383 0 0 0 63717 141 0 0 25 0 5 0 498436490 1168404480 224309 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 285255 224309 198 36 0 283160 0
[pid=29827/tid=29841] ppid=29825 vsize=1141020 CPUtime=158.59 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 36841 0 0 0 15807 52 0 0 25 0 5 0 498436810 1168404480 224309 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=1141020 CPUtime=158.63 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 63540 0 0 0 15848 15 0 0 25 0 5 0 498436810 1168404480 224309 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=1141020 CPUtime=159.06 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 39141 0 0 0 15858 48 0 0 25 0 5 0 498436810 1168404480 224309 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=1141020 CPUtime=159.07 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 83852 0 0 0 15882 25 0 0 25 0 5 0 498436810 1168404480 224309 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 638.58
Current children cumulated vsize (KiB) 1141020

[startup+222.301 s]
/proc/loadavg: 8.03 7.96 7.96 9/186 29849
/proc/meminfo: memFree=24417616/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=1276752 CPUtime=878.22 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 258553 0 0 0 87647 175 0 0 25 0 5 0 498436490 1307394048 258479 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 319188 258479 198 36 0 317093 0
[pid=29827/tid=29841] ppid=29825 vsize=1276752 CPUtime=218.58 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 42701 0 0 0 21793 65 0 0 25 0 5 0 498436810 1307394048 258480 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=1276752 CPUtime=218.28 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 73527 0 0 0 21810 18 0 0 25 0 5 0 498436810 1307394048 258480 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=1276752 CPUtime=219.05 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 43873 0 0 0 21845 60 0 0 25 0 5 0 498436810 1307394048 258480 33554432000 4194304 4338100 140733326829008 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=1276752 CPUtime=219.07 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 97444 0 0 0 21877 30 0 0 25 0 5 0 498436810 1307394048 258480 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 878.22
Current children cumulated vsize (KiB) 1276752

################
# More data... #
################

/proc/meminfo: memFree=22744300/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=2376344 CPUtime=3034.04 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 533791 0 0 0 302967 437 0 0 25 0 5 0 498436490 2433376256 533717 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 594086 533717 198 36 0 591991 0
[pid=29827/tid=29841] ppid=29825 vsize=2376344 CPUtime=757.12 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 83609 0 0 0 75548 164 0 0 25 0 5 0 498436810 2433376256 533717 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=2376344 CPUtime=756.14 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 158217 0 0 0 75565 49 0 0 25 0 5 0 498436810 2433376256 533717 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=2376344 CPUtime=758.71 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 83076 0 0 0 75717 154 0 0 25 0 5 0 498436810 2433376256 533717 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=2376344 CPUtime=758.84 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 207880 0 0 0 75816 68 0 0 25 0 5 0 498436810 2433376256 533717 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3034.04
Current children cumulated vsize (KiB) 2376344

[startup+822.3 s]
/proc/loadavg: 8.12 8.09 8.01 9/186 29874
/proc/meminfo: memFree=22618792/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=2423776 CPUtime=3273.66 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 545941 0 0 0 326909 457 0 0 25 0 5 0 498436490 2481946624 545867 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 605944 545867 198 36 0 603849 0
[pid=29827/tid=29841] ppid=29825 vsize=2423776 CPUtime=817.1 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 84373 0 0 0 81539 171 0 0 25 0 5 0 498436810 2481946624 545867 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=2423776 CPUtime=815.82 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 162320 0 0 0 81531 51 0 0 25 0 5 0 498436810 2481946624 545867 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=2423776 CPUtime=818.7 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 84536 0 0 0 81708 162 0 0 25 0 5 0 498436810 2481946624 545867 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=2423776 CPUtime=818.82 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 213703 0 0 0 81810 72 0 0 25 0 5 0 498436810 2481946624 545867 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3273.66
Current children cumulated vsize (KiB) 2423776

[startup+882.301 s]
/proc/loadavg: 8.09 8.08 8.01 9/186 29876
/proc/meminfo: memFree=22509512/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=2524128 CPUtime=3513.31 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 570991 0 0 0 350850 481 0 0 25 0 5 0 498436490 2584707072 570917 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 631032 570917 198 36 0 628937 0
[pid=29827/tid=29841] ppid=29825 vsize=2524128 CPUtime=877.1 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 89574 0 0 0 87530 180 0 0 25 0 5 0 498436810 2584707072 570917 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=2524128 CPUtime=875.48 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 170229 0 0 0 87494 54 0 0 25 0 5 0 498436810 2584707072 570917 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=2524128 CPUtime=878.7 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 89968 0 0 0 87699 171 0 0 25 0 5 0 498436810 2584707072 570917 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=2524128 CPUtime=878.81 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 220211 0 0 0 87806 75 0 0 25 0 5 0 498436810 2584707072 570917 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3513.31
Current children cumulated vsize (KiB) 2524128

[startup+942.301 s]
/proc/loadavg: 8.08 8.08 8.01 9/186 29878
/proc/meminfo: memFree=22444132/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=2577560 CPUtime=3752.95 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 584868 0 0 0 374795 500 0 0 25 0 5 0 498436490 2639421440 584794 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 644390 584794 198 36 0 642295 0
[pid=29827/tid=29841] ppid=29825 vsize=2577560 CPUtime=936.98 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 92178 0 0 0 93512 186 0 0 25 0 5 0 498436810 2639421440 584794 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=2577560 CPUtime=935.24 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 173491 0 0 0 93468 56 0 0 25 0 5 0 498436810 2639421440 584794 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=2577560 CPUtime=938.68 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 92318 0 0 0 93689 179 0 0 25 0 5 0 498436810 2639421440 584794 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=2577560 CPUtime=938.82 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 225872 0 0 0 93804 78 0 0 25 0 5 0 498436810 2639421440 584794 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3752.95
Current children cumulated vsize (KiB) 2577560

[startup+1002.3 s]
/proc/loadavg: 8.12 8.09 8.02 9/186 29879
/proc/meminfo: memFree=22318444/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=2621608 CPUtime=3992.57 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 595573 0 0 0 398741 516 0 0 25 0 5 0 498436490 2684526592 595499 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 655402 595499 198 36 0 653307 0
[pid=29827/tid=29841] ppid=29825 vsize=2621608 CPUtime=996.63 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 95656 0 0 0 99469 194 0 0 25 0 5 0 498436810 2684526592 595499 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=2621608 CPUtime=995.24 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 177402 0 0 0 99466 58 0 0 25 0 5 0 498436810 2684526592 595499 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=2621608 CPUtime=998.66 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 92409 0 0 0 99683 183 0 0 25 0 5 0 498436810 2684526592 595499 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=2621608 CPUtime=998.8 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 229097 0 0 0 99801 79 0 0 25 0 5 0 498436810 2684526592 595499 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3992.57
Current children cumulated vsize (KiB) 2621608

[startup+1062.3 s]
/proc/loadavg: 8.12 8.10 8.02 9/186 29881
/proc/meminfo: memFree=22182104/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=2712448 CPUtime=4232.25 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 618180 0 0 0 422690 535 0 0 25 0 5 0 498436490 2777546752 618106 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 678112 618106 198 36 0 676017 0
[pid=29827/tid=29841] ppid=29825 vsize=2712448 CPUtime=1056.31 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 99713 0 0 0 105429 202 0 0 25 0 5 0 498436810 2777546752 618106 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=2712448 CPUtime=1055.25 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 183687 0 0 0 105465 60 0 0 25 0 5 0 498436810 2777546752 618106 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=2712448 CPUtime=1058.66 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 96996 0 0 0 105676 190 0 0 25 0 5 0 498436810 2777546752 618106 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=2712448 CPUtime=1058.81 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 236775 0 0 0 105799 82 0 0 25 0 5 0 498436810 2777546752 618106 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4232.25
Current children cumulated vsize (KiB) 2712448

[startup+1122.3 s]
/proc/loadavg: 8.08 8.10 8.02 9/186 29882
/proc/meminfo: memFree=22008328/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=2743984 CPUtime=4471.93 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 626636 0 0 0 446644 549 0 0 25 0 5 0 498436490 2809839616 626562 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 685996 626562 198 36 0 683901 0
[pid=29827/tid=29841] ppid=29825 vsize=2743984 CPUtime=1115.98 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 101145 0 0 0 111392 206 0 0 25 0 5 0 498436810 2809839616 626562 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=2743984 CPUtime=1115.24 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 187155 0 0 0 111462 62 0 0 25 0 5 0 498436810 2809839616 626562 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=2743984 CPUtime=1118.67 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 98115 0 0 0 111671 196 0 0 25 0 5 0 498436810 2809839616 626562 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=2743984 CPUtime=1118.81 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 239212 0 0 0 111797 84 0 0 25 0 5 0 498436810 2809839616 626562 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4471.93
Current children cumulated vsize (KiB) 2743984

[startup+1182.3 s]
/proc/loadavg: 8.03 8.08 8.02 9/186 29884
/proc/meminfo: memFree=21813100/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=2778648 CPUtime=4711.59 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 636932 0 0 0 470596 563 0 0 25 0 5 0 498436490 2845335552 636858 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 694662 636858 198 36 0 692567 0
[pid=29827/tid=29841] ppid=29825 vsize=2778648 CPUtime=1175.65 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 103425 0 0 0 117353 212 0 0 25 0 5 0 498436810 2845335552 636858 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=2778648 CPUtime=1175.23 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 188428 0 0 0 117461 62 0 0 25 0 5 0 498436810 2845335552 636858 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=2778648 CPUtime=1178.66 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 100010 0 0 0 117666 200 0 0 25 0 5 0 498436810 2845335552 636858 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254347 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=2778648 CPUtime=1178.8 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 244060 0 0 0 117794 86 0 0 25 0 5 0 498436810 2845335552 636858 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4711.59
Current children cumulated vsize (KiB) 2778648



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.51 s]
/proc/loadavg: 8.08 8.09 8.02 9/186 29885
/proc/meminfo: memFree=21809100/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=2784904 CPUtime=4800.31 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) S 29825 29827 29500 0 -1 4202496 637949 0 0 0 479465 566 0 0 25 0 5 0 498436490 2851741696 637875 33554432000 4194304 4338100 140733326829008 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 696226 637875 198 36 0 694131 0
[pid=29827/tid=29841] ppid=29825 vsize=2784904 CPUtime=1197.74 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 103442 0 0 0 119560 214 0 0 25 0 5 0 498436810 2851741696 637875 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=29827/tid=29842] ppid=29825 vsize=2784904 CPUtime=1197.45 cores=1,3,5,7
/proc/29827/task/29842/stat : 29842 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 189397 0 0 0 119682 63 0 0 25 0 5 0 498436810 2851741696 637875 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=29827/tid=29843] ppid=29825 vsize=2784904 CPUtime=1200.88 cores=1,3,5,7
/proc/29827/task/29843/stat : 29843 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 100027 0 0 0 119886 202 0 0 25 0 5 0 498436810 2851741696 637875 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=29827/tid=29844] ppid=29825 vsize=2784904 CPUtime=1201.01 cores=1,3,5,7
/proc/29827/task/29844/stat : 29844 (ttsth-5-4) R 29825 29827 29500 0 -1 4202560 244074 0 0 0 120015 86 0 0 25 0 5 0 498436810 2851741696 637875 33554432000 4194304 4338100 140733326829008 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.31
Current children cumulated vsize (KiB) 2784904

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.61 s]
/proc/loadavg: 8.08 8.09 8.02 9/186 29885
/proc/meminfo: memFree=21809100/32951124 swapFree=67111524/67111528
[pid=29827] ppid=29825 vsize=0 CPUtime=4800.42 cores=1,3,5,7
/proc/29827/stat : 29827 (ttsth-5-4) Z 29825 29827 29500 0 -1 4203548 637949 0 0 0 479467 575 0 0 18 0 2 0 498436490 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/29827/statm: 0 0 0 0 0 0 0
[pid=29827/tid=29841] ppid=29825 vsize=0 CPUtime=1197.83 cores=1,3,5,7
/proc/29827/task/29841/stat : 29841 (ttsth-5-4) R 29825 29827 29500 0 -1 4203588 103442 0 0 0 119560 223 0 0 25 0 2 0 498436810 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.42
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1204.63
CPU time (s): 4800.45
CPU user time (s): 4794.68
CPU system time (s): 5.77512
CPU usage (%): 398.499
Max. virtual memory (cumulated for all children) (KiB): 2784904

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4794.68
system time used= 5.77512
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 637949
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2128
involuntary context switches= 33184

runsolver used 2.01969 second user time and 4.56331 second system time

The end

Launcher Data 

Begin job on node142 at 2011-04-29 08:56:50
IDJOB=3274156
IDBENCH=83154
IDSOLVER=1654
FILE ID=node142/3274156-1304060210
RUNJOBID= node142-1304060210-29816
PBS_JOBID= 13170434
Free space on /tmp= 71316 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n240-s78945233-sat.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3274156-1304060210/watcher-3274156-1304060210 -o /tmp/evaluation-result-3274156-1304060210/solver-3274156-1304060210 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3274156-1304060210.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1f90ac4288669aff970e47f4201b5ea4
RANDOM SEED=1940039724

node142.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5131.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      26594080 kB
Buffers:       1652972 kB
Cached:        4101128 kB
SwapCached:          4 kB
Active:         682680 kB
Inactive:      5135540 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      26594080 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             168 kB
Writeback:           0 kB
AnonPages:       64512 kB
Mapped:          14476 kB
Slab:           475632 kB
PageTables:       4888 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   184624 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71316 MiB
End job on node142 at 2011-04-29 09:16:57