Trace number 3263630

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.25 1202.24

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_60_20_50_3-dir.shuffled-as.sat03-114.cnf
MD5SUM9ffa0325c9e00b41c025f41d20ec5a0c
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark4252.27
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses49070
Sum of the clauses size98700
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 249050
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 520

Solver Data 

0.01/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263630-1304097280.cnf 
0.01/0.03	c Using SATTIMEOUT of 4800
0.01/0.03	c Using SATRAM of 15500
0.01/0.03	c Size of subset node = 32
0.01/0.03	c Variables = 600
0.01/0.03	c Clauses = 49070
0.01/0.03	c Literals = 98700
0.01/0.03	c Reduced size:
0.01/0.03	c Variables = 600
0.01/0.03	c Clauses = 49070
0.01/0.03	c Literals = 98700
0.01/0.03	c a2
0.01/0.06	c a1
0.09/0.13	c Independent sub-problems = 1
0.09/0.13	c Sub-problem with 600 variables and 49070 clauses
0.09/0.13	c using permutation method 4
0.09/0.13	c count = 1
0.09/0.13	c Ordering finished 0.120 seconds
0.09/0.13	c mapped 0.120 seconds
0.09/0.13	c permuted 0.120 seconds
0.09/0.13	c about to sort
0.09/0.13	c sorted
0.19/0.25	c clauses now 49070
0.19/0.25	c init finished 0.140 seconds
0.19/0.25	c tree built 0.240 seconds
0.19/0.25	c 
0.19/0.25	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.25	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.25	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.25	c ................................ | ..........                  1 |                                  0.2         4.2             
0.19/0.25	c ................................ | ..........                  2 |                                  0.2         8.3             
0.19/0.25	c ................................ | ..........                  3 |                                  0.2        12.5             
0.19/0.25	c ................................ | ..........                  4 |                                  0.2        16.7             
0.19/0.25	c ................................ | ..........                  5 |                                  0.2        20.8             
0.19/0.25	c ................................ | ..........               1000 | 5            42                  0.2      4000.0         2081
95.91/24.22	c ................................ | ..........            4649000 | 93687        593654             96.0     48417.0     13146116
191.76/48.26	c ................................ | ..........            7137000 | 143715       1025527           192.0     37169.9     19797805
287.65/72.29	c ................................ | ..........            8927000 | 190749       1416583           288.0     30995.5     24281986
383.92/96.32	c ................................ | ..........           10224000 | 237803       1687554           384.0     26625.0     30144835
479.78/120.35	c ................................ | *.........           11418000 | 282824       1950822           480.0     23787.0     35750361
576.02/144.40	c ................................ | *.........           13271000 | 335497       2224267           576.0     23038.7     37874076
671.88/168.45	c ................................ | *.........           15224000 | 381653       2490413           672.1     22652.1     42265364
767.73/192.48	c ................................ | *.........           16659000 | 423247       2723398           768.1     21689.4     45458844
863.99/216.52	c ................................ | *.........           18110000 | 454340       2930461           864.1     20958.7     47530262
959.86/240.54	c ................................ | **........           19195000 | 495150       3106804           960.0     19994.4     49872864
1056.13/264.64	c ................................ | **........           20500000 | 529180       3299258          1056.0     19412.5     54093365
1151.96/288.62	c ................................ | **........           21855000 | 564848       3481691          1152.0     18970.9     55335334
1247.83/312.65	c ................................ | **........           22806000 | 594983       3640901          1248.0     18273.5     55460463
1343.70/336.68	c ................................ | **........           23907000 | 634489       3836271          1344.0     17787.5     57070608
1439.95/360.72	c ................................ | ***.......           24936000 | 670736       4077517          1440.0     17316.3     58509086
1535.81/384.75	c ................................ | ***.......           25951000 | 709736       4268151          1536.0     16894.9     59822443
1631.67/408.80	c ................................ | ***.......           26721000 | 746852       4404651          1632.1     16372.5     61470302
1727.92/432.83	c ................................ | ***.......           27558000 | 783195       4584551          1728.0     15947.5     62547039
1823.78/456.89	c ................................ | ***.......           28381000 | 821959       4720927          1824.1     15558.6     65459529
1920.01/480.90	c ................................ | ****......           29184000 | 854812       4956738          1920.0     15199.8     68557529
2015.83/504.96	c ................................ | ****......           30265000 | 888161       5162688          2016.1     15011.7     69044556
2111.68/528.99	c ................................ | ****......           31318000 | 918195       5382496          2112.1     14828.2     70701484
2207.94/553.05	c ................................ | ****......           32470000 | 953131       5578096          2208.1     14704.7     72312979
2303.79/577.06	c ................................ | ****......           33532000 | 995351       5739686          2304.1     14553.5     75571895
2399.64/601.09	c ................................ | *****.....           34373000 | 1036612      5905141          2400.0     14322.0     77488199
2495.91/625.14	c ................................ | *****.....           35319000 | 1075274      6050554          2496.1     14149.8     80676811
2591.76/649.16	c ................................ | *****.....           36356000 | 1107375      6198327          2592.0     14026.2     83291400
2687.95/673.21	c ................................ | *****.....           38027000 | 1152562      6401887          2688.0     14146.8     84951043
2783.80/697.25	c ................................ | *****.....           39788000 | 1182835      6655348          2784.0     14291.5     86522012
2880.02/721.30	c ................................ | ******....           41172000 | 1215553      6882086          2880.0     14295.6     87265461
2975.86/745.34	c ................................ | ******....           42446000 | 1246534      7104577          2976.0     14262.6     87812333
3071.67/769.38	c ................................ | ******....           43599000 | 1283379      7287587          3072.0     14192.2     87812333
3167.92/793.42	c ................................ | ******....           44769000 | 1316495      7456073          3168.0     14131.6     88460733
3263.73/817.48	c ................................ | ******....           45876000 | 1358685      7643253          3264.1     14054.8     88645545
3359.99/841.50	c ................................ | *******...           47147000 | 1396672      7822618          3360.0     14031.8     89185764
3455.80/865.55	c ................................ | *******...           48730000 | 1436209      8010249          3456.0     14100.0     89599619
3552.04/889.61	c ................................ | *******...           50397000 | 1470624      8223671          3552.1     14188.0     91786962
3647.93/913.63	c ................................ | *******...           51729000 | 1498877      8416962          3648.0     14179.9     93367105
3743.78/937.66	c ................................ | *******...           53017000 | 1532926      8606829          3744.0     14160.4     94422678
3839.63/961.69	c ................................ | ********..           54107000 | 1554405      8798336          3840.0     14090.3     95514323
3935.87/985.73	c ................................ | ********..           55242000 | 1572435      8985730          3936.0     14035.1     96557702
4031.70/1009.77	c ................................ | ********..           56394000 | 1596676      9216657          4032.0     13986.6     97420658
4127.96/1033.82	c ................................ | ********..           57425000 | 1619396      9470816          4128.1     13910.9     97474273
4223.81/1057.85	c ................................ | ********..           58420000 | 1641594      9654633          4224.0     13830.5     98307911
4319.64/1081.89	c ................................ | *********.           59361000 | 1666523      9836999          4320.0     13740.9     98307911
4415.87/1105.95	c ................................ | *********.           60515000 | 1693534      10017638         4416.1     13703.2     98307911
4511.73/1129.97	c ................................ | *********.           61318000 | 1710003      10155223         4512.0     13589.9     98348713
4607.98/1154.02	c ................................ | *********.           62298000 | 1733517      10298758         4608.1     13519.4     99785483
4703.84/1178.05	c ................................ | *********.           63194000 | 1751285      10411262         4704.1     13434.0    101462807
4799.72/1202.09	c CPU limit (4800) reached
4799.72/1202.09	c Time = 4800.09 seconds
4799.72/1202.09	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263630-1304097280/watcher-3263630-1304097280 -o /tmp/evaluation-result-3263630-1304097280/solver-3263630-1304097280 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263630-1304097280.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.11 7.64 7.69 2/172 12960
/proc/meminfo: memFree=26620300/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=9540 CPUtime=0 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) R 12958 12960 12928 0 -1 4202496 468 0 0 0 0 0 0 0 25 0 1 0 521326657 9768960 410 33554432000 4194304 4338100 140733893574000 18446744073709551615 242970211013 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/12960/statm: 2385 411 147 36 0 290 0

[startup+0.022732 s]
/proc/loadavg: 6.11 7.64 7.69 2/172 12960
/proc/meminfo: memFree=26620300/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=12984 CPUtime=0.01 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) R 12958 12960 12928 0 -1 4202496 1292 0 0 0 1 0 0 0 25 0 1 0 521326657 13295616 1234 33554432000 4194304 4338100 140733893574000 18446744073709551615 242970211013 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/12960/statm: 3246 1236 148 36 0 1151 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 12984

[startup+0.100726 s]
/proc/loadavg: 6.11 7.64 7.69 2/172 12960
/proc/meminfo: memFree=26620300/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=19200 CPUtime=0.09 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) R 12958 12960 12928 0 -1 4202496 3882 0 0 0 9 0 0 0 25 0 1 0 521326657 19660800 2797 33554432000 4194304 4338100 140733893574000 18446744073709551615 4296613 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/12960/statm: 4800 2799 155 36 0 2705 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 19200

[startup+0.310775 s]
/proc/loadavg: 6.11 7.64 7.69 2/172 12960
/proc/meminfo: memFree=26620300/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=239976 CPUtime=2.49 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 14861 0 0 0 246 3 0 0 25 0 5 0 521326657 245735424 13584 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 77160 15350 176 36 0 75065 0
Current children cumulated CPU time (s) 2.49
Current children cumulated vsize (KiB) 239976

[startup+1.15651 s]
/proc/loadavg: 6.11 7.64 7.69 2/172 12960
/proc/meminfo: memFree=26620300/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=315028 CPUtime=3.85 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 17270 0 0 0 381 4 0 0 25 0 5 0 521326657 322588672 15993 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 78757 15993 176 36 0 76662 0
Current children cumulated CPU time (s) 3.85
Current children cumulated vsize (KiB) 315028

[startup+1.55644 s]
/proc/loadavg: 6.11 7.64 7.69 2/172 12960
/proc/meminfo: memFree=26620300/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=327804 CPUtime=5.45 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 20320 0 0 0 538 7 0 0 25 0 5 0 521326657 335671296 19043 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 81951 19043 176 36 0 79856 0
Current children cumulated CPU time (s) 5.45
Current children cumulated vsize (KiB) 327804

[startup+3.10117 s]
/proc/loadavg: 6.11 7.64 7.69 10/184 12975
/proc/meminfo: memFree=26419468/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=368996 CPUtime=11.62 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 30650 0 0 0 1146 16 0 0 25 0 5 0 521326657 377851904 29374 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 92249 29374 176 36 0 90154 0
[pid=12960/tid=12968] ppid=12958 vsize=368996 CPUtime=2.84 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 7582 0 0 0 282 2 0 0 25 0 5 0 521326683 377851904 29374 33554432000 4194304 4338100 140733893574000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=368996 CPUtime=2.84 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 4267 0 0 0 279 5 0 0 25 0 5 0 521326683 377851904 29374 33554432000 4194304 4338100 140733893574000 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=368996 CPUtime=2.84 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 6212 0 0 0 283 1 0 0 24 0 5 0 521326683 377851904 29374 33554432000 4194304 4338100 140733893574000 18446744073709551615 242969945530 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=368996 CPUtime=2.83 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 2919 0 0 0 277 6 0 0 24 0 5 0 521326683 377851904 29374 33554432000 4194304 4338100 140733893574000 18446744073709551615 4256989 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.62
Current children cumulated vsize (KiB) 368996

[startup+6.30062 s]
/proc/loadavg: 6.34 7.67 7.70 10/184 12975
/proc/meminfo: memFree=26302932/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=435740 CPUtime=24.41 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 47554 0 0 0 2403 38 0 0 25 0 5 0 521326657 446197760 46277 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 108935 46277 176 36 0 106840 0
[pid=12960/tid=12968] ppid=12958 vsize=435740 CPUtime=6.04 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 13266 0 0 0 600 4 0 0 25 0 5 0 521326683 446197760 46277 33554432000 4194304 4338100 140733893574000 18446744073709551615 4230441 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=435740 CPUtime=6.03 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 7293 0 0 0 590 13 0 0 25 0 5 0 521326683 446197760 46277 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254840 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=435740 CPUtime=6.04 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 12049 0 0 0 602 2 0 0 25 0 5 0 521326683 446197760 46277 33554432000 4194304 4338100 140733893574000 18446744073709551615 4256288 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=435740 CPUtime=6.01 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 5275 0 0 0 586 15 0 0 25 0 5 0 521326683 446197760 46277 33554432000 4194304 4338100 140733893574000 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.41
Current children cumulated vsize (KiB) 435740

[startup+12.7005 s]
/proc/loadavg: 6.47 7.67 7.70 10/184 12975
/proc/meminfo: memFree=26094468/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=537332 CPUtime=49.99 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 72933 0 0 0 4924 75 0 0 25 0 5 0 521326657 550227968 71656 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 134333 71656 176 36 0 132238 0
[pid=12960/tid=12968] ppid=12958 vsize=537332 CPUtime=12.44 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 22489 0 0 0 1236 8 0 0 25 0 5 0 521326683 550227968 71660 33554432000 4194304 4338100 140733893574000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=537332 CPUtime=12.44 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 12173 0 0 0 1215 29 0 0 25 0 5 0 521326683 550227968 71660 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=537332 CPUtime=12.43 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 19363 0 0 0 1238 5 0 0 25 0 5 0 521326683 550227968 71660 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=537332 CPUtime=12.4 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 9241 0 0 0 1209 31 0 0 25 0 5 0 521326683 550227968 71660 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 49.99
Current children cumulated vsize (KiB) 537332

[startup+25.5003 s]
/proc/loadavg: 6.85 7.72 7.71 10/184 12976
/proc/meminfo: memFree=25751140/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=724016 CPUtime=101.1 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 120433 0 0 0 9972 138 0 0 25 0 5 0 521326657 741392384 119156 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 181004 119156 176 36 0 178909 0
[pid=12960/tid=12968] ppid=12958 vsize=724016 CPUtime=25.24 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 42145 0 0 0 2510 14 0 0 25 0 5 0 521326683 741392384 119156 33554432000 4194304 4338100 140733893574000 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=724016 CPUtime=25.22 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 18704 0 0 0 2468 54 0 0 25 0 5 0 521326683 741392384 119156 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=724016 CPUtime=25.16 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 34701 0 0 0 2508 8 0 0 25 0 5 0 521326683 741392384 119156 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=724016 CPUtime=25.2 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 15212 0 0 0 2461 59 0 0 25 0 5 0 521326683 741392384 119156 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 101.1
Current children cumulated vsize (KiB) 724016

[startup+51.1009 s]
/proc/loadavg: 7.41 7.77 7.73 9/184 12976
/proc/meminfo: memFree=25240484/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=944524 CPUtime=203.34 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 176944 0 0 0 20074 260 0 0 25 0 5 0 521326657 967192576 175667 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 236131 175667 176 36 0 234036 0
[pid=12960/tid=12968] ppid=12958 vsize=944524 CPUtime=50.83 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 64479 0 0 0 5062 21 0 0 25 0 5 0 521326683 967192576 175667 33554432000 4194304 4338100 140733893574000 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=944524 CPUtime=50.82 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 26344 0 0 0 4975 107 0 0 25 0 5 0 521326683 967192576 175667 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=944524 CPUtime=50.63 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 55566 0 0 0 5049 14 0 0 25 0 5 0 521326683 967192576 175667 33554432000 4194304 4338100 140733893574000 18446744073709551615 242969945484 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=944524 CPUtime=50.79 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 20884 0 0 0 4964 115 0 0 25 0 5 0 521326683 967192576 175667 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 203.34
Current children cumulated vsize (KiB) 944524

[startup+102.301 s]
/proc/loadavg: 7.79 7.82 7.75 9/184 12978
/proc/meminfo: memFree=24450716/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=1300852 CPUtime=407.87 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 267600 0 0 0 40342 445 0 0 25 0 5 0 521326657 1332072448 266323 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 325213 266323 176 36 0 323118 0
[pid=12960/tid=12968] ppid=12958 vsize=1300852 CPUtime=102.03 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 94970 0 0 0 10169 34 0 0 25 0 5 0 521326683 1332072448 266323 33554432000 4194304 4338100 140733893574000 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=1300852 CPUtime=102.01 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 39628 0 0 0 10013 188 0 0 25 0 5 0 521326683 1332072448 266323 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=1300852 CPUtime=101.58 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 90627 0 0 0 10134 24 0 0 25 0 5 0 521326683 1332072448 266323 33554432000 4194304 4338100 140733893574000 18446744073709551615 4229520 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=1300852 CPUtime=101.97 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 32704 0 0 0 10001 196 0 0 25 0 5 0 521326683 1332072448 266323 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 407.87
Current children cumulated vsize (KiB) 1300852

[startup+162.301 s]
/proc/loadavg: 7.97 7.86 7.76 9/184 12979
/proc/meminfo: memFree=23743036/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=1637008 CPUtime=647.54 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 353260 0 0 0 64120 634 0 0 25 0 5 0 521326657 1676296192 351983 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 409252 351983 176 36 0 407157 0
[pid=12960/tid=12968] ppid=12958 vsize=1637008 CPUtime=162.01 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 133893 0 0 0 16152 49 0 0 25 0 5 0 521326683 1676296192 351983 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=1637008 CPUtime=162 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 46265 0 0 0 15933 267 0 0 25 0 5 0 521326683 1676296192 351983 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=1637008 CPUtime=161.29 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 123724 0 0 0 16096 33 0 0 25 0 5 0 521326683 1676296192 351983 33554432000 4194304 4338100 140733893574000 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=1637008 CPUtime=161.96 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 39707 0 0 0 15914 282 0 0 25 0 5 0 521326683 1676296192 351983 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 647.54
Current children cumulated vsize (KiB) 1637008

[startup+222.3 s]
/proc/loadavg: 7.99 7.89 7.78 10/184 12981
/proc/meminfo: memFree=23183408/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=1863056 CPUtime=887.14 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 410982 0 0 0 87894 820 0 0 25 0 5 0 521326657 1907769344 409705 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 465764 409705 176 36 0 463669 0
[pid=12960/tid=12968] ppid=12958 vsize=1863056 CPUtime=221.97 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 161453 0 0 0 22135 62 0 0 25 0 5 0 521326683 1907769344 409705 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0

################
# More data... #
################

/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 80316 0 0 0 69067 1093 0 0 25 0 5 0 521326683 3183456256 730480 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2804.17
Current children cumulated vsize (KiB) 3108844

[startup+762.3 s]
/proc/loadavg: 7.79 7.98 7.87 10/184 12995
/proc/meminfo: memFree=20018624/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=3160316 CPUtime=3043.72 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 746901 0 0 0 301845 2527 0 0 25 0 5 0 521326657 3236163584 745624 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 790079 745624 176 36 0 787984 0
[pid=12960/tid=12968] ppid=12958 vsize=3160316 CPUtime=760.15 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 317301 0 0 0 75864 151 0 0 25 0 5 0 521326683 3236163584 745624 33554432000 4194304 4338100 140733893574000 18446744073709551615 242970548283 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=3160316 CPUtime=761.59 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 92661 0 0 0 75015 1144 0 0 25 0 5 0 521326683 3236163584 745624 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=3160316 CPUtime=760.18 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 245254 0 0 0 75943 75 0 0 25 0 5 0 521326683 3236163584 745624 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=3160316 CPUtime=761.51 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 82014 0 0 0 74997 1154 0 0 25 0 5 0 521326683 3236163584 745624 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3043.72
Current children cumulated vsize (KiB) 3160316

[startup+822.301 s]
/proc/loadavg: 7.92 7.98 7.88 9/184 12997
/proc/meminfo: memFree=19676740/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=3242572 CPUtime=3283.32 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 759557 0 0 0 325664 2668 0 0 25 0 5 0 521326657 3320393728 758280 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 810643 758280 176 36 0 808548 0
[pid=12960/tid=12968] ppid=12958 vsize=3242572 CPUtime=820.13 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 325584 0 0 0 81854 159 0 0 25 0 5 0 521326683 3320393728 758280 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=3242572 CPUtime=821.52 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 94248 0 0 0 80940 1212 0 0 25 0 5 0 521326683 3320393728 758280 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=3242572 CPUtime=820.16 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 246059 0 0 0 81941 75 0 0 25 0 5 0 521326683 3320393728 758280 33554432000 4194304 4338100 140733893574000 18446744073709551615 4285738 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=3242572 CPUtime=821.22 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 83995 0 0 0 80903 1219 0 0 25 0 5 0 521326683 3320393728 758280 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3283.32
Current children cumulated vsize (KiB) 3242572

[startup+882.301 s]
/proc/loadavg: 8.02 7.99 7.89 9/184 12999
/proc/meminfo: memFree=19310428/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=3324072 CPUtime=3522.9 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 781061 0 0 0 349458 2832 0 0 25 0 5 0 521326657 3403849728 779784 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 831018 779784 176 36 0 828923 0
[pid=12960/tid=12968] ppid=12958 vsize=3324072 CPUtime=880.1 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 345169 0 0 0 87839 171 0 0 25 0 5 0 521326683 3403849728 779784 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=3324072 CPUtime=881.49 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 94783 0 0 0 86863 1286 0 0 25 0 5 0 521326683 3403849728 779784 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=3324072 CPUtime=880.14 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 246923 0 0 0 87938 76 0 0 25 0 5 0 521326683 3403849728 779784 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=3324072 CPUtime=880.9 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 84515 0 0 0 86793 1297 0 0 25 0 5 0 521326683 3403849728 779784 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3522.9
Current children cumulated vsize (KiB) 3324072

[startup+942.306 s]
/proc/loadavg: 8.00 7.99 7.90 9/184 13000
/proc/meminfo: memFree=19021872/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=3433592 CPUtime=3762.57 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 810923 0 0 0 373287 2970 0 0 25 0 5 0 521326657 3515998208 809646 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 858398 809646 176 36 0 856303 0
[pid=12960/tid=12968] ppid=12958 vsize=3433592 CPUtime=939.82 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 368741 0 0 0 93800 182 0 0 25 0 5 0 521326683 3515998208 809646 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=3433592 CPUtime=941.47 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 95369 0 0 0 92801 1346 0 0 25 0 5 0 521326683 3515998208 809646 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=3433592 CPUtime=940.14 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 252039 0 0 0 93935 79 0 0 25 0 5 0 521326683 3515998208 809646 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=3433592 CPUtime=940.86 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 85103 0 0 0 92726 1360 0 0 25 0 5 0 521326683 3515998208 809646 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3762.57
Current children cumulated vsize (KiB) 3433592

[startup+1002.3 s]
/proc/loadavg: 8.00 7.99 7.90 9/184 13002
/proc/meminfo: memFree=18685196/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=3527472 CPUtime=4002.15 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 835603 0 0 0 397122 3093 0 0 25 0 5 0 521326657 3612131328 834326 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 881868 834326 176 36 0 879773 0
[pid=12960/tid=12968] ppid=12958 vsize=3527472 CPUtime=999.59 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 387362 0 0 0 99767 192 0 0 25 0 5 0 521326683 3612131328 834326 33554432000 4194304 4338100 140733893574000 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=3527472 CPUtime=1001.46 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 95960 0 0 0 98745 1401 0 0 25 0 5 0 521326683 3612131328 834326 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=3527472 CPUtime=1000 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 256933 0 0 0 99919 81 0 0 25 0 5 0 521326683 3612131328 834326 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=3527472 CPUtime=1000.81 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 85677 0 0 0 98666 1415 0 0 25 0 5 0 521326683 3612131328 834326 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4002.15
Current children cumulated vsize (KiB) 3527472

[startup+1062.3 s]
/proc/loadavg: 8.00 7.99 7.91 9/184 13003
/proc/meminfo: memFree=18529940/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=3558924 CPUtime=4241.76 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 846560 0 0 0 420962 3214 0 0 25 0 5 0 521326657 3644338176 845283 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 889731 845283 176 36 0 887636 0
[pid=12960/tid=12968] ppid=12958 vsize=3558924 CPUtime=1059.57 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 395675 0 0 0 105758 199 0 0 25 0 5 0 521326683 3644338176 845283 33554432000 4194304 4338100 140733893574000 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=3558924 CPUtime=1061.42 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 96692 0 0 0 104685 1457 0 0 25 0 5 0 521326683 3644338176 845283 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=3558924 CPUtime=1059.7 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 257648 0 0 0 105888 82 0 0 25 0 5 0 521326683 3644338176 845283 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=3558924 CPUtime=1060.77 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 86874 0 0 0 104605 1472 0 0 25 0 5 0 521326683 3644338176 845283 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4241.76
Current children cumulated vsize (KiB) 3558924

[startup+1122.3 s]
/proc/loadavg: 8.07 8.02 7.92 9/184 13005
/proc/meminfo: memFree=18308464/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=3561960 CPUtime=4481.36 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 849480 0 0 0 444804 3332 0 0 25 0 5 0 521326657 3647447040 848203 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 890490 848203 176 36 0 888395 0
[pid=12960/tid=12968] ppid=12958 vsize=3561960 CPUtime=1119.53 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 396816 0 0 0 111748 205 0 0 25 0 5 0 521326683 3647447040 848203 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=3561960 CPUtime=1121.4 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 97078 0 0 0 110624 1516 0 0 25 0 5 0 521326683 3647447040 848203 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=3561960 CPUtime=1119.42 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 258333 0 0 0 111859 83 0 0 25 0 5 0 521326683 3647447040 848203 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=3561960 CPUtime=1120.73 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 87582 0 0 0 110548 1525 0 0 25 0 5 0 521326683 3647447040 848203 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4481.36
Current children cumulated vsize (KiB) 3561960

[startup+1182.3 s]
/proc/loadavg: 8.17 8.06 7.94 9/184 13007
/proc/meminfo: memFree=18007252/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=3670028 CPUtime=4721.03 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) S 12958 12960 12928 0 -1 4202496 877906 0 0 0 468651 3452 0 0 25 0 5 0 521326657 3758108672 876629 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 917507 876630 176 36 0 915412 0
[pid=12960/tid=12968] ppid=12958 vsize=3670028 CPUtime=1179.51 cores=1,3,5,7
/proc/12960/task/12968/stat : 12968 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 414397 0 0 0 117736 215 0 0 25 0 5 0 521326683 3758108672 876630 33554432000 4194304 4338100 140733893574000 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12960/tid=12969] ppid=12958 vsize=3670028 CPUtime=1181.38 cores=1,3,5,7
/proc/12960/task/12969/stat : 12969 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 97346 0 0 0 116571 1567 0 0 25 0 5 0 521326683 3758108672 876630 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12960/tid=12970] ppid=12958 vsize=3670028 CPUtime=1179.14 cores=1,3,5,7
/proc/12960/task/12970/stat : 12970 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 268648 0 0 0 117827 87 0 0 25 0 5 0 521326683 3758108672 876630 33554432000 4194304 4338100 140733893574000 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=3670028 CPUtime=1180.71 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4202560 87845 0 0 0 116491 1580 0 0 25 0 5 0 521326683 3758108672 876630 33554432000 4194304 4338100 140733893574000 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4721.03
Current children cumulated vsize (KiB) 3670028



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.1 s]
/proc/loadavg: 8.12 8.06 7.94 6/181 13007
/proc/meminfo: memFree=18105056/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) Z 12958 12960 12928 0 -1 4203548 885012 0 0 0 476525 3485 0 0 18 0 2 0 521326657 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 0 0 0 0 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=0 CPUtime=1200.5 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4203588 87902 0 0 0 118455 1595 0 0 25 0 2 0 521326683 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.21 s]
/proc/loadavg: 8.12 8.06 7.94 6/181 13007
/proc/meminfo: memFree=18105056/32951132 swapFree=67111364/67111528
[pid=12960] ppid=12958 vsize=0 CPUtime=4800.21 cores=1,3,5,7
/proc/12960/stat : 12960 (ttsth-5-4) Z 12958 12960 12928 0 -1 4203548 885012 0 0 0 476525 3496 0 0 18 0 2 0 521326657 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/12960/statm: 0 0 0 0 0 0 0
[pid=12960/tid=12971] ppid=12958 vsize=0 CPUtime=1200.61 cores=1,3,5,7
/proc/12960/task/12971/stat : 12971 (ttsth-5-4) R 12958 12960 12928 0 -1 4203588 87902 0 0 0 118455 1606 0 0 25 0 2 0 521326683 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.21
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.24
CPU time (s): 4800.25
CPU user time (s): 4765.26
CPU system time (s): 34.9867
CPU usage (%): 399.276
Max. virtual memory (cumulated for all children) (KiB): 3695976

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4765.26
system time used= 34.9867
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 885012
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 8557
involuntary context switches= 31977

runsolver used 1.69274 second user time and 3.89741 second system time

The end

Launcher Data 

Begin job on node128 at 2011-04-29 19:14:40
IDJOB=3263630
IDBENCH=83988
IDSOLVER=1654
FILE ID=node128/3263630-1304097280
RUNJOBID= node128-1304097280-12947
PBS_JOBID= 13170611
Free space on /tmp= 66972 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_60_20_50_3-dir.shuffled-as.sat03-114.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263630-1304097280/watcher-3263630-1304097280 -o /tmp/evaluation-result-3263630-1304097280/solver-3263630-1304097280 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263630-1304097280.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 9ffa0325c9e00b41c025f41d20ec5a0c
RANDOM SEED=1156877426

node128.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5434.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      26620704 kB
Buffers:       1281604 kB
Cached:        4391788 kB
SwapCached:        160 kB
Active:         339592 kB
Inactive:      5401920 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      26620704 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:             952 kB
Writeback:           8 kB
AnonPages:       67620 kB
Mapped:          14304 kB
Slab:           523960 kB
PageTables:       4692 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   191000 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66972 MiB
End job on node128 at 2011-04-29 19:34:44