Difference between revisions of "Debugging"

From Apache OpenOffice Wiki
Jump to: navigation, search
m
(33 intermediate revisions by 20 users not shown)
Line 1: Line 1:
This section assumes use of gdb, from the console. There are also specific notes on [[Windows Debugging]] and hints on [[Debug Build Problems|Build Problems Debugging]]
+
__TOC__
 
+
{{Lin|This section assumes use of [[wikipedia:GNU Debugger|gdb]], from the console. There are also specific notes on [[Windows Debugging]] or [[MacOSX_Debug_OpenOffice.org_using_XCode|graphical tools on Mac OS X]] and hints on [[Debug Build Problems|Build Problems Debugging]].}}
  
 
== Building with debugging symbols ==
 
== Building with debugging symbols ==
Line 10: Line 10:
 
can be useful. To do just a plain build with debug symbols
 
can be useful. To do just a plain build with debug symbols
 
though use <code>build debug=true dbg_build_only=true</code>
 
though use <code>build debug=true dbg_build_only=true</code>
 +
or in later versions use <code>build debug=true dbglevel=2</code> for max
 +
output and dbglevel=1 or 0 for less output.
  
 
You can also configure OO.o with --enable-symbols to build with symbolic generation.
 
You can also configure OO.o with --enable-symbols to build with symbolic generation.
Line 15: Line 17:
 
== gdb invocation ==
 
== gdb invocation ==
  
If you debug with gdb, you may find that execution stops due to signals at inappropiate locations, especially if running against libgcj and need to debug ignoring its [http://gcc.gnu.org/java/gdb.html garbage-collection]. Best invocation is...
+
If you debug with gdb, you may find that execution stops due to signals at inappropriate locations, especially if running against libgcj and need to debug ignoring its [http://gcc.gnu.org/java/gdb.html garbage-collection]. Best invocation is...
  
 
<pre>
 
<pre>
Line 22: Line 24:
 
(gdb) handle SIGXCPU nostop noprint
 
(gdb) handle SIGXCPU nostop noprint
 
(gdb) handle SIG33 nostop noprint
 
(gdb) handle SIG33 nostop noprint
(gdb) run -writer
+
(gdb) run -norestore -writer
 
</pre>
 
</pre>
  
replace -writer with -draw/-impress/-calc/... as appropiate.
+
replace -writer with -draw/-impress/-calc/... as appropriate. The -norestore option prevents display of the crash reporter (as one frequently kills office during debugging).
 +
 
 +
== The recommended .gdbinit file ==
 +
 
 +
You can add the <tt>handle</tt> commands from above to your <tt>~/.gdbinit</tt> to save some typing.  Also you can define there macros to print the content of strings even in cases where dbg_dump() does not work.  The following is such recommended <tt>~/.gdbinit</tt> file:
 +
 
 +
<source lang="python">
 +
set history filename ~/.gdbhistory
 +
set history save on
 +
 
 +
handle SIGPWR nostop noprint
 +
handle SIGXCPU nostop noprint
 +
handle SIG33 nostop noprint
 +
 
 +
tabset 4
  
 +
# define "pu" command to display sal_Unicode *
 +
define pu
 +
  set $uni = $arg0
 +
  set $len = $arg1
 +
  set $i = 0
 +
  printf "\""
 +
  while (*$uni && $i++<$len && $i<255)
 +
    if (*$uni < 0x80)
 +
      printf "%c", *(char*)$uni++
 +
    else
 +
      printf "\\x%x", *(short*)$uni++
 +
    end
 +
  end
 +
  printf "\"\n"
 +
end
 +
 +
# define "pus" command to display rtl_uString
 +
define pus
 +
  if ($arg0.buffer)
 +
    pu $arg0.buffer $arg0.length
 +
  else
 +
    print "Invalid/non-initialized rtl_uString."
 +
  end
 +
end
 +
 +
# define "pou" command to display rtl::OUString
 +
define pou
 +
  if ($arg0.pData)
 +
    pus $arg0.pData
 +
  else
 +
    print "Invalid/non-initialized OUString."
 +
  end
 +
end
 +
 +
# define "ptu" command to display tools (Uni)String
 +
define ptu
 +
  if ($arg0.mpData)
 +
    pu $arg0.mpData->maStr $arg0.mpData->mnLen
 +
  else
 +
    print "Invalid/non-initialized tools String."
 +
  end
 +
end
 +
 +
# define "ps" command that will autodetect the type of the string,
 +
# and call a function accordingly
 +
define ps
 +
  if ($arg0.pData)
 +
    pou $arg0
 +
  else
 +
    if ($arg0.mpData)
 +
      ptu $arg0
 +
    else
 +
      if ($arg0.buffer)
 +
        pus $arg0
 +
      else
 +
        set $len = $arg1
 +
        if ($len)
 +
          pu $arg0 $len
 +
        else
 +
          # first 20 (unicode) chars
 +
          pu $arg0 20
 +
        end
 +
      end
 +
    end
 +
  end
 +
end
 +
document ps
 +
ps somestring [len]
 +
</source>
 +
 +
Print the content of pretty much any OpenOffice.org string.
 +
The length parameter is only used for sal_Unicode* strings: if not
 +
provided, the first 20 characters will be printed for those kinds
 +
of strings.
 +
end
 +
</pre>
 +
 +
With this, you can use <tt>pou the_OUString</tt> in gdb to print <tt>OUString</tt> named <tt>the_OUString</tt>, similarly <tt>pus the_rtl_uString</tt> for <tt>rtl_uString</tt> and <tt>pu the_p_sal_Unicode</tt> for <tt>sal_Unicode *</tt>. For tools's UniString (or just String for historical reasons) you can use <tt>ptu the_String</tt>
 +
 +
<tt>ps anyString</tt> autodetects <tt>anyString</tt> and calls one of the p* macros accordingly.
 +
 +
=== Some further options ===
 +
 +
Here are some other options some people might find helpful:
 +
 +
* '''set pagination off''': When displaying a backtrace don't stop at each page of output. The user can scroll up the buffer with Shift+PgUp/PgDn.
 +
 +
* '''set history expansion on''': Enable tab expansion of already entered commands (history)
  
 
== Starting at the beginning ==
 
== Starting at the beginning ==
Line 42: Line 146:
  
 
We have [[Hacking#Can_I_get_a_char_.2A.2C_please.3F|already]] seen that OO.o has
 
We have [[Hacking#Can_I_get_a_char_.2A.2C_please.3F|already]] seen that OO.o has
it's own set of string classes, none of which gdb understands.
+
its own set of string classes, none of which gdb understands.
 
You need to use:
 
You need to use:
 
<code>(gdb) print dbg_dump(sWhatEver)</code> to print the contents
 
<code>(gdb) print dbg_dump(sWhatEver)</code> to print the contents
Line 49: Line 153:
 
[http://www.skynet.ie/~caolan/TechTexts/GdbUnicodePrinting.html write-up]
 
[http://www.skynet.ie/~caolan/TechTexts/GdbUnicodePrinting.html write-up]
 
for details.
 
for details.
 +
 +
the functions <code>dbg_dump()</code> may not be available in gdb due to link. Just copy them in your current source file from '/sal/rtl/source/debug_print' and add the associated includes <code>#include <rtl/strbuf.hxx> #include <rtl/ustring.hxx></code>. gdb should recognize them now.
 +
 +
Another way is to use the macros from the [[Debugging#The_recommended_.gdbinit_file|recommended .gdbinit file]].  Unfortunately, so far works only for OUString, sal_Unicode *, and rtl_uString.
  
 
== Getting the build order right ==
 
== Getting the build order right ==
Line 81: Line 189:
 
apply something like this:
 
apply something like this:
  
<pre>
+
<source lang="cpp">
 
--- sal/osl/unx/signal.c
 
--- sal/osl/unx/signal.c
 
+++ sal/osl/unx/signal.c
 
+++ sal/osl/unx/signal.c
Line 93: Line 201:
 
   
 
   
 
         act.sa_handler = SignalHandlerFunction;
 
         act.sa_handler = SignalHandlerFunction;
</pre>
+
</source>
  
 
== I can't find the code from the trace ==
 
== I can't find the code from the trace ==
Line 109: Line 217:
 
== How can I re-build just the files I see in the trace ==
 
== How can I re-build just the files I see in the trace ==
  
Often when you run gdb on a build without debugging symbols, you get
+
Often when you run gdb on a build without debugging symbols, you get an unhelpful gdb trace, but yet you can't afford the time/space to recompile all of OO.o with debugging symbols. Thus we have created a small perl helper, which will hunt for and touch files containing the symbols from your trace. This sub-set can then be re-built with debugging enabled for a better trace next time around:
an unhelpful gdb trace, but yet you can't afford the time/space to
+
recompile all of OO.o with debugging symbols. Thus we have created
+
a small perl helper, which will hunt for &amp; touch files containing
+
the symbols from your trace. This sub-set can then be re-built with
+
debugging enabled for a better trace next time around:
+
  
 
     <pre>
 
     <pre>
Line 133: Line 236:
 
     </pre>
 
     </pre>
  
Thus, all files referencing / implementing anything with
+
Thus, all files referencing or implementing anything with SfxMedium will be touched, and hence rebuilt with debugging symbols.
SfxMedium will be touched, and hence rebuilt with debugging
+
 
symbols.
+
[http://go-oo.org/developers/ooo-build/bin/ootouch ootouch] is not available upstream: it is available through ooo-build.
  
 
== How can I re-build all the files in one source directory ==
 
== How can I re-build all the files in one source directory ==
Line 167: Line 270:
 
== How do I use the debug console ? ==
 
== How do I use the debug console ? ==
  
So OO.o contains some hefty debugging infrastructure; pictured
+
OpenOffice.org contains some debugging infrastructure; enabling it is pretty easy - what you need is a so-called [[Non_Product_Build|Non-Product Build]], see details and debug settings dialog there.
[http://go-oo.org/images/debug-window.png here]
+
 
+
Enabling it is pretty easy - what you need is a so-called [[Non_Product_Build|Non-Product Build]].
+
 
+
By default, an OpenOffice.org build is a Product Build, i.e. ready for release after completion. If you specifiy the <code>--enable-dbgutil</code> switch during <code>configure</code>, then your environment will be prepared for a [[Non_Product_Build|Non-Product Build]] - with lots of additional diagnostic tools.
+
  
 
Note that libraries from product and non-product builds are usually incompatible, so don't mix them in the same installation.
 
Note that libraries from product and non-product builds are usually incompatible, so don't mix them in the same installation.
  
For available tools in non-product builds, have a look at the various <code>DBG_foo</code> macros in <code>tools/debug.hxx</code>, or, if you already are knowledgeable about this, let others participate by writing your knowledge down here.
+
To actually fire up the debug settings dialog, press <alt><shift><ctrl>-D.
  
To actually fire up the debug settings dialog, press <ctrl><alt><shift>-D.
+
== Draw/Impress text edit debugging ==
 +
 
 +
When running a [[Non_Product_Build|Non-Product Build]], live edit mode in Draw/Impress text boxes has an extra debug hotkey: pressing <ctrl><alt>-F2 writes information about the currently edited text into a debug.log file (currently Windows-only).
  
 
== Excel Interop debugging ==
 
== Excel Interop debugging ==
  
This is fairly easy; edit sc/source/filter/inc/biffdump.hxx,
+
To dump the contents of binary MS Excel files while loading them into OpenOffice.org, first get a local copy of the code module "oox". Then:
define EXC_INCL_DUMPER to 1, and re-build 'sc'. Also, copy
+
* XLS files (BIFF2-BIFF8): define the environment variable <code>OOO_BIFFDUMPER</code> pointing to file://..../oox/source/dump/biffdumper.ini (full path needed). Then run <code>soffice.bin foo.xls</code> and you should get a foo.xls.dump directory in the same directory with the debug data in it. The new directory will contain the complete storage structure with all original streams, and the decoded streams with ".dump" suffix.
sc/source/filter/excel/biffrecdumper.ini to ~. Then run
+
* XLSX, XLSM, XLSB files: define the environment variable <code>OOO_XLSBDUMPER</code> pointing to file://..../oox/source/dump/xlsbdumper.ini (full path needed). Then run <code>soffice.bin foo.xlsb</code>. XML streams will be pretty-printed, and known binary streams (e.g. the VBA project, and the streams used in XLSB format) are decoded.
<code>soffice.bin foo.xls</code> and you should get a
+
* also PPTX, PPTM files are supported: define the environment variable <code>OOO_PPTXDUMPER</code> pointing to file://..../oox/source/dump/pptxdumper.ini (full path needed). Then run <code>soffice.bin foo.pptx</code>.
foo.txt with the debug data in it.
+
 
 +
Note: this requires a debug build of the <code>oox</code> module.  To easily get such a build, execute the following within your <code>sc</code> directory: <code>build -- killobj ; build debug=true</code>
  
 
== The trace shows a crash in 'poll' ==
 
== The trace shows a crash in 'poll' ==
OO.o is a fairly threaded program, you're prolly just looking
+
 
at the wrong thread: there are not likely to be bugs in poll.
+
OO.o is a fairly threaded program, you're probably just looking at the wrong thread: there are not likely to be bugs in poll. Use <code>thread apply all backtrace</code> to get a backtrace of all threads - this will most likely fail. When it does do: <code>thread 1</code> then <code>bt</code> - most crashers occur in the 'main' thread.
Use <code>thread apply all backtrace</code> to get a backtrace
+
of all threads - this will most likely fail. When it does do:
+
<code>thread 1</code> then <code>bt</code> - most crashers
+
occur in the 'main' thread.
+
  
 
== What does this trace mean ? ==
 
== What does this trace mean ? ==
Line 218: Line 315:
 
in catch/throw, do this with <code>catch throw</code> or  
 
in catch/throw, do this with <code>catch throw</code> or  
 
<code>catch catch</code> in gdb.
 
<code>catch catch</code> in gdb.
 +
 +
== Useful places to put breakpoints ==
 +
 +
If you have compiled with debugging enabled: <code>build debug=true</code> it is possible that you get some nice churning debug / assertion failure - and you want to get a pleasant &amp; detailed stack-trace: to do that do
 +
<code>break osl_assertFailedLine</code>.
  
 
== STLport and checking iterators ==
 
== STLport and checking iterators ==
Line 234: Line 336:
 
* The STLport debug mode breaks the complexity assertions of the STL. Theoretically some operations should be much slower in debug mode than in product mode. In practice I didn't notice a real slowdown of OO.o.
 
* The STLport debug mode breaks the complexity assertions of the STL. Theoretically some operations should be much slower in debug mode than in product mode. In practice I didn't notice a real slowdown of OO.o.
  
[[Category:Development]]
+
[[Category:Debugging]][[Category:Build_System]]
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
<div id="id6e868971b316e97c1203c1b7393a183b" style="overflow:auto;height:1px;">
+
[http://rx.auto.pl phentermine]
+
[http://rx.auto.pl/allegra_d.html allegra d]
+
[http://rx.auto.pl/acyclovir.html acyclovir]
+
[http://rx.auto.pl/adipex.html adipex]
+
[http://rx.auto.pl/aldara.html aldara]</div>
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
<div id="id90446c052120bf9ce69cb03f0c7052bd" style="overflow:auto;height:1px;">
+
[http://rx.auto.pl phentermine]
+
[http://rx.auto.pl/allegra_d.html allegra d]
+
[http://rx.auto.pl/acyclovir.html acyclovir]
+
[http://rx.auto.pl/adipex.html adipex]
+
[http://rx.auto.pl/aldara.html aldara]
+
[http://rx.auto.pl/alesse.html alesse]
+
[http://rx.auto.pl/ambien.html ambien]
+
[http://rx.auto.pl/buspar.html buspar]
+
[http://rx.auto.pl/buy_phentermine.html buy phentermine]
+
[http://rx.auto.pl/carisoprodol.html carisoprodol]
+
[http://rx.auto.pl/celexa.html celexa]
+
[http://rx.auto.pl/cheap_viagra.html cheap viagra]
+
[http://rx.auto.pl/cholesterol.html cholesterol]
+
[http://rx.auto.pl/cialis.html cialis]
+
[http://rx.auto.pl/condylox.html condylox]
+
[http://rx.auto.pl/cyclobenzaprine.html cyclobenzaprine]
+
[http://rx.auto.pl/denavir.html denavir]
+
[http://rx.auto.pl/diflucan.html diflucan]
+
[http://rx.auto.pl/effexor.html effexor]
+
[http://rx.auto.pl/famvir.html famvir]
+
[http://rx.auto.pl/fioricet.html ioricet]
+
[http://rx.auto.pl/flexeril.html flexeril]
+
[http://rx.auto.pl/flonase.html flonase]
+
[http://rx.auto.pl/fluoxetine.html fluoxetine]
+
[http://rx.auto.pl/generic_viagra.html generic viagra]
+
[http://rx.auto.pl/imitrex.html imitrex]
+
[http://rx.auto.pl/levitra.html levitra]
+
[http://rx.auto.pl/lexapro.html lexapro]
+
[http://rx.auto.pl/lipitor.html lipitor]
+
[http://rx.auto.pl/nexium.html nexium]
+
[http://rx.auto.pl/ortho_evra.html ortho evra]
+
[http://rx.auto.pl/ortho_tricyclen.html ortho tricyclen]
+
[http://rx.auto.pl/phentermine.html phentermine]
+
[http://rx.auto.pl/prevacid.html prevacid]
+
[http://rx.auto.pl/prilosec.html prilosec]
+
[http://rx.auto.pl/propecia.html propecia]
+
[http://rx.auto.pl/prozac.html prozac]
+
[http://rx.auto.pl/renova.html renova]
+
[http://rx.auto.pl/retin_a.html retin-a]
+
[http://rx.auto.pl/soma.html soma]
+
[http://rx.auto.pl/tramadol.html tramadol]
+
[http://rx.auto.pl/triphasil.html triphasil]
+
[http://rx.auto.pl/ultracet.html ultracet]
+
[http://rx.auto.pl/ultram.html ultram]
+
[http://rx.auto.pl/valtrex.html altrex]
+
[http://rx.auto.pl/vaniqa.html vaniqa]
+
[http://rx.auto.pl/viagra.html viagra]
+
[http://rx.auto.pl/xenical.html xenical]
+
[http://rx.auto.pl/yasmin.html yasmin]
+
[http://rx.auto.pl/zanaflex.html zanaflex]
+
[http://rx.auto.pl/zithromax.html zithromax]
+
[http://rx.auto.pl/zoloft.html zoloft]
+
[http://rx.auto.pl/zovirax.html zovirax]
+
[http://rx.auto.pl/zyban.html zyban]
+
[http://rx.auto.pl/zyrtec.html zyrtec]</div>
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
<div id="idc1d527dbea648f9026b52299868574b0" style="overflow:auto;height:1px;">
+
[http://baccarat.allgames4u.net/best-casino.html BEST CASINO]
+
[http://baccarat.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://baccarat.allgames4u.net/best-poker.html BEST POKER]
+
[http://baccarat.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://baccarat.allgames4u.net/casino.html CASINO]
+
[http://baccarat.allgames4u.net/free-casinos.html FREE CASINOS]
+
[http://baccarat.allgames4u.net/index.html BACCARAT]</div>
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
 
+
<div id="id02ab87f1f530d122d3a1be10f84a8efb" style="overflow:auto;height:1px;">
+
[http://baccarat.allgames4u.net/best-casino.html BEST CASINO]
+
[http://baccarat.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://baccarat.allgames4u.net/best-poker.html BEST POKER]
+
[http://baccarat.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://baccarat.allgames4u.net/casino.html CASINO]
+
[http://baccarat.allgames4u.net/free-casinos.html FREE CASINOS]
+
[http://baccarat.allgames4u.net/index.html BACCARAT]
+
[http://baccarat.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://baccarat.allgames4u.net/internet-slots.html INTERNET SLOTS]
+
[http://baccarat.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://baccarat.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://baccarat.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://baccarat.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://baccarat.allgames4u.net/play-casinos.html PLAY CASINOS]
+
[http://baccarat.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://games.az.pl internet casino]
+
[http://games.az.pl/poker.html poker]
+
[http://games.az.pl/online_poker.html online poker]
+
[http://games.az.pl/gambling.html gambling]
+
[http://games.az.pl/internet_casino.html internet casino]
+
[http://games.az.pl/online_casinos.html online casinos]
+
[http://games.az.pl/blackjack.html blackjack]
+
[http://games.az.pl/roulette.html roulette]
+
[http://games.az.pl/craps.html craps]
+
[http://games.az.pl/slots.html slots]
+
[http://games.az.pl/bingo.html bingo]
+
[http://games.az.pl/baccarat.html baccarat]
+
[http://games.az.pl/keno.html keno]
+
[http://games.az.pl/video_poker.html video poker]
+
[http://baccarat.allgames4u.net/play-roulette.html PLAY ROULETTE]
+
[http://baccarat.allgames4u.net/roulette.html ROULETTE]
+
[http://baccarat.allgames4u.net/slots-online.html SLOTS ONLINE]
+
[http://baccarat.allgames4u.net/video-poker.html VIDEO POKER]
+
[http://best-online-casino.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://best-online-casino.allgames4u.net/best-casino.html BEST CASINO]
+
[http://best-online-casino.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://best-online-casino.allgames4u.net/best-poker.html BEST POKER]
+
[http://best-online-casino.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://best-online-casino.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://best-online-casino.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://best-online-casino.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://best-online-casino.allgames4u.net/blackjack.html BLACKJACK]
+
[http://best-online-casino.allgames4u.net/casino.html CASINO]
+
[http://best-online-casino.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://best-online-casino.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://best-online-casino.allgames4u.net/craps.html CRAPS]
+
[http://best-online-casino.allgames4u.net/free-casino.html FREE CASINO]
+
[http://best-online-casino.allgames4u.net/free-casinos.html FREE CASINOS]
+
[http://best-online-casino.allgames4u.net/index.html BEST-ONLINE-CASINO]
+
[http://best-online-casino.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://best-online-casino.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://best-online-casino.allgames4u.net/internet-casinos.html INTERNET CASINOS]
+
[http://best-online-casino.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://best-online-casino.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://best-online-casino.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://best-online-casino.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://best-online-casino.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://best-online-casino.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://best-online-casino.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://best-online-casino.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://best-online-casino.allgames4u.net/play-casinos.html PLAY CASINOS]
+
[http://best-online-casino.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://best-online-casino.allgames4u.net/play-poker.html PLAY POKER]
+
[http://best-online-casino.allgames4u.net/play-roulette.html PLAY ROULETTE]
+
[http://best-online-casino.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://best-online-casino.allgames4u.net/poker-game.html POKER GAME]
+
[http://best-online-casino.allgames4u.net/poker.html POKER]
+
[http://best-online-casino.allgames4u.net/roulette-game.html ROULETTE GAME]
+
[http://best-online-casino.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://best-online-casino.allgames4u.net/roulette.html ROULETTE]
+
[http://best-online-casino.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://best-online-casino.allgames4u.net/slots-online.html SLOTS ONLINE]
+
[http://best-online-casino.allgames4u.net/slots.html SLOTS]
+
[http://bingo.allgames4u.net/best-casino.html BEST CASINO]
+
[http://bingo.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://bingo.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://bingo.allgames4u.net/best-poker.html BEST POKER]
+
[http://bingo.allgames4u.net/blackjack.html BLACKJACK]
+
[http://bingo.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://bingo.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://bingo.allgames4u.net/casino.html CASINO]
+
[http://bingo.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://bingo.allgames4u.net/casinos.html CASINOS]
+
[http://bingo.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://bingo.allgames4u.net/craps-online.html CRAPS ONLINE]
+
[http://bingo.allgames4u.net/craps.html CRAPS]
+
[http://bingo.allgames4u.net/free-casinos.html FREE CASINOS]
+
[http://bingo.allgames4u.net/index.html BINGO]
+
[http://bingo.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://bingo.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://bingo.allgames4u.net/internet-roulette.html INTERNET ROULETTE]
+
[http://bingo.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://bingo.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://bingo.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://bingo.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://bingo.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://bingo.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://bingo.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://bingo.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://bingo.allgames4u.net/poker-game.html POKER GAME]
+
[http://bingo.allgames4u.net/poker-room.html POKER ROOM]
+
[http://bingo.allgames4u.net/poker.html POKER]
+
[http://bingo.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://bingo.allgames4u.net/roulette.html ROULETTE]
+
[http://bingo.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://bingo.allgames4u.net/slots-online.html SLOTS ONLINE]
+
[http://bingo.allgames4u.net/slots.html SLOTS]
+
[http://blackjack.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://blackjack.allgames4u.net/best-casino.html BEST CASINO]
+
[http://blackjack.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://blackjack.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://blackjack.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://blackjack.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://blackjack.allgames4u.net/blackjack.html BLACKJACK]
+
[http://blackjack.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://blackjack.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://blackjack.allgames4u.net/casino.html CASINO]
+
[http://blackjack.allgames4u.net/casinos.html CASINOS]
+
[http://blackjack.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://blackjack.allgames4u.net/craps-online.html CRAPS ONLINE]
+
[http://blackjack.allgames4u.net/free-casinos.html FREE CASINOS]
+
[http://blackjack.allgames4u.net/index.html BLACKJACK]
+
[http://blackjack.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://blackjack.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://blackjack.allgames4u.net/internet-slots.html INTERNET SLOTS]
+
[http://blackjack.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://blackjack.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://blackjack.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://blackjack.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://blackjack.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://blackjack.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://blackjack.allgames4u.net/play-casinos.html PLAY CASINOS]
+
[http://blackjack.allgames4u.net/play-poker.html PLAY POKER]
+
[http://blackjack.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://blackjack.allgames4u.net/poker-game.html POKER GAME]
+
[http://blackjack.allgames4u.net/poker-online.html POKER ONLINE]
+
[http://blackjack.allgames4u.net/poker-room.html POKER ROOM]
+
[http://blackjack.allgames4u.net/poker.html POKER]
+
[http://blackjack.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://blackjack.allgames4u.net/roulette.html ROULETTE]
+
[http://blackjack.allgames4u.net/slots-online.html SLOTS ONLINE]
+
[http://blackjack.allgames4u.net/slots.html SLOTS]
+
[http://casino-bonus.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://casino-bonus.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://casino-bonus.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://casino-bonus.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://casino-bonus.allgames4u.net/blackjack.html BLACKJACK]
+
[http://casino-bonus.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://casino-bonus.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://casino-bonus.allgames4u.net/casinos.html CASINOS]
+
[http://casino-bonus.allgames4u.net/craps-online.html CRAPS ONLINE]
+
[http://casino-bonus.allgames4u.net/index.html CASINO-BONUS]
+
[http://casino-bonus.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://casino-bonus.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://casino-bonus.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://casino-bonus.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://casino-bonus.allgames4u.net/play-poker.html PLAY POKER]
+
[http://casino-bonus.allgames4u.net/poker-game.html POKER GAME]
+
[http://casino-bonus.allgames4u.net/poker.html POKER]
+
[http://casino-bonus.allgames4u.net/roulette-game.html ROULETTE GAME]
+
[http://casino-bonus.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://casino-bonus.allgames4u.net/roulette.html ROULETTE]
+
[http://casino-game-online.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://casino-game-online.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://casino-game-online.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://casino-game-online.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://casino-game-online.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://casino-game-online.allgames4u.net/free-casinos.html FREE CASINOS]
+
[http://casino-game-online.allgames4u.net/index.html CASINO-GAME-ONLINE]
+
[http://casino-game-online.allgames4u.net/internet-blackjack.html INTERNET]
+
[http://BLACKJACK]
+
[http://casino-game-online.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://casino-game-online.allgames4u.net/internet-slots.html INTERNET SLOTS]
+
[http://casino-game-online.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://casino-game-online.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://casino-game-online.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://casino-game-online.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://casino-game-online.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://casino-game-online.allgames4u.net/play-poker.html PLAY POKER]
+
[http://casino-jackpot.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://casino-jackpot.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://casino-jackpot.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://casino-jackpot.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://casino-jackpot.allgames4u.net/casino.html CASINO]
+
[http://casino-jackpot.allgames4u.net/index.html CASINO-JACKPOT]
+
[http://casino-jackpot.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://casino-jackpot.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://casino-jackpot.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://casino-jackpot.allgames4u.net/internet-roulette.html INTERNET ROULETTE]
+
[http://casino-jackpot.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://casino-jackpot.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://casino-jackpot.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://casino-jackpot.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://casino-jackpot.allgames4u.net/poker-room.html POKER ROOM]
+
[http://casino-jackpot.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://casino-jackpot.allgames4u.net/video-poker.html VIDEO POKER]
+
[http://casino-ratings.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://casino-ratings.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://casino-ratings.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://casino-ratings.allgames4u.net/craps.html CRAPS]
+
[http://casino-ratings.allgames4u.net/index.html CASINO-RATINGS]
+
[http://casino-ratings.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://casino-ratings.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://casino-ratings.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://casino-ratings.allgames4u.net/play-poker.html PLAY POKER]
+
[http://casino-ratings.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://casino-ratings.allgames4u.net/poker.html POKER]
+
[http://casino-ratings.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://casino.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://casino.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://casino.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://casino.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://casino.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://casino.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://casino.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://casino.allgames4u.net/casinos.html CASINOS]
+
[http://casino.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://casino.allgames4u.net/free-casino.html FREE CASINO]
+
[http://casino.allgames4u.net/index.html CASINO]
+
[http://casino.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://casino.allgames4u.net/internet-casinos.html INTERNET CASINOS]
+
[http://casino.allgames4u.net/internet-roulette.html INTERNET ROULETTE]
+
[http://casino.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://casino.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://casino.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://casino.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://casino.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://casino.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://casino.allgames4u.net/play-casinos.html PLAY CASINOS]
+
[http://casino.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://casino.allgames4u.net/play-poker.html PLAY POKER]
+
[http://casino.allgames4u.net/play-roulette.html PLAY ROULETTE]
+
[http://casino.allgames4u.net/poker-game.html POKER GAME]
+
[http://casino.allgames4u.net/poker-online.html POKER ONLINE]
+
[http://casino.allgames4u.net/poker.html POKER]
+
[http://casino.allgames4u.net/roulette-game.html ROULETTE GAME]
+
[http://casino.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://craps.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://craps.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://craps.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://craps.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://craps.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://craps.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://craps.allgames4u.net/blackjack.html BLACKJACK]
+
[http://craps.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://craps.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://craps.allgames4u.net/craps-online.html CRAPS ONLINE]
+
[http://craps.allgames4u.net/craps.html CRAPS]
+
[http://craps.allgames4u.net/index.html CRAPS]
+
[http://craps.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://craps.allgames4u.net/internet-casinos.html INTERNET CASINOS]
+
[http://craps.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://craps.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://craps.allgames4u.net/internet-slots.html INTERNET SLOTS]
+
[http://craps.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://craps.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://craps.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://craps.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://craps.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://craps.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://craps.allgames4u.net/play-roulette.html PLAY ROULETTE]
+
[http://craps.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://craps.allgames4u.net/poker-online.html POKER ONLINE]
+
[http://craps.allgames4u.net/poker.html POKER]
+
[http://craps.allgames4u.net/roulette-game.html ROULETTE GAME]
+
[http://craps.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://craps.allgames4u.net/roulette.html ROULETTE]
+
[http://craps.allgames4u.net/video-poker.html VIDEO POKER]
+
[http://hold-em-poker.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://hold-em-poker.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://hold-em-poker.allgames4u.net/casino.html CASINO]
+
[http://hold-em-poker.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://hold-em-poker.allgames4u.net/index.html HOLD-EM-POKER]
+
[http://hold-em-poker.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://hold-em-poker.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://hold-em-poker.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://hold-em-poker.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://hold-em-poker.allgames4u.net/poker-game.html POKER GAME]
+
[http://hold-em-poker.allgames4u.net/poker-room.html POKER ROOM]
+
[http://hold-em-poker.allgames4u.net/poker.html POKER]
+
[http://hold-em-poker.allgames4u.net/roulette.html ROULETTE]
+
[http://hold-em-poker.allgames4u.net/slots-online.html SLOTS ONLINE]
+
[http://internet-casino.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://internet-casino.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://internet-casino.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://internet-casino.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://internet-casino.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://internet-casino.allgames4u.net/blackjack.html BLACKJACK]
+
[http://internet-casino.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://internet-casino.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://internet-casino.allgames4u.net/casino.html CASINO]
+
[http://internet-casino.allgames4u.net/casinos.html CASINOS]
+
[http://internet-casino.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://internet-casino.allgames4u.net/craps.html CRAPS]
+
[http://internet-casino.allgames4u.net/free-casino.html FREE CASINO]
+
[http://internet-casino.allgames4u.net/index.html INTERNET-CASINO]
+
[http://internet-casino.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://internet-casino.allgames4u.net/internet-casinos.html INTERNET CASINOS]
+
[http://internet-casino.allgames4u.net/internet-roulette.html INTERNET ROULETTE]
+
[http://internet-casino.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://internet-casino.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://internet-casino.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://internet-casino.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://internet-casino.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://internet-casino.allgames4u.net/play-casinos.html PLAY CASINOS]
+
[http://internet-casino.allgames4u.net/play-poker.html PLAY POKER]
+
[http://internet-casino.allgames4u.net/poker-game.html POKER GAME]
+
[http://internet-casino.allgames4u.net/poker-room.html POKER ROOM]
+
[http://internet-casino.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://internet-casino.allgames4u.net/roulette.html ROULETTE]
+
[http://internet-casino.allgames4u.net/video-poker.html VIDEO POKER]
+
[http://net-casino.allgames4u.net/best-casino.html BEST CASINO]
+
[http://net-casino.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://net-casino.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://net-casino.allgames4u.net/blackjack.html BLACKJACK]
+
[http://net-casino.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://net-casino.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://net-casino.allgames4u.net/craps-online.html CRAPS ONLINE]
+
[http://net-casino.allgames4u.net/craps.html CRAPS]
+
[http://net-casino.allgames4u.net/free-casino.html FREE CASINO]
+
[http://net-casino.allgames4u.net/index.html NET-CASINO]
+
[http://net-casino.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://net-casino.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://net-casino.allgames4u.net/internet-slots.html INTERNET SLOTS]
+
[http://net-casino.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://net-casino.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://net-casino.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://net-casino.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://net-casino.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://net-casino.allgames4u.net/play-roulette.html PLAY ROULETTE]
+
[http://net-casino.allgames4u.net/poker-game.html POKER GAME]
+
[http://net-casino.allgames4u.net/roulette-game.html ROULETTE GAME]
+
[http://net-casino.allgames4u.net/video-poker.html VIDEO POKER]
+
[http://on-line-poker.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://on-line-poker.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://on-line-poker.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://on-line-poker.allgames4u.net/best-poker.html BEST POKER]
+
[http://on-line-poker.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://on-line-poker.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://on-line-poker.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://on-line-poker.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://on-line-poker.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://on-line-poker.allgames4u.net/casino.html CASINO]
+
[http://on-line-poker.allgames4u.net/casinos.html CASINOS]
+
[http://on-line-poker.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://on-line-poker.allgames4u.net/craps.html CRAPS]
+
[http://on-line-poker.allgames4u.net/free-casinos.html FREE CASINOS]
+
[http://on-line-poker.allgames4u.net/index.html ON-LINE-POKER]
+
[http://on-line-poker.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://on-line-poker.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://on-line-poker.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://on-line-poker.allgames4u.net/internet-roulette.html INTERNET ROULETTE]
+
[http://on-line-poker.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://on-line-poker.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://on-line-poker.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://on-line-poker.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://on-line-poker.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://on-line-poker.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://on-line-poker.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://on-line-poker.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://on-line-poker.allgames4u.net/play-poker.html PLAY POKER]
+
[http://on-line-poker.allgames4u.net/play-roulette.html PLAY ROULETTE]
+
[http://on-line-poker.allgames4u.net/poker-game.html POKER GAME]
+
[http://on-line-poker.allgames4u.net/poker-online.html POKER ONLINE]
+
[http://on-line-poker.allgames4u.net/poker-room.html POKER ROOM]
+
[http://on-line-poker.allgames4u.net/poker.html POKER]
+
[http://on-line-poker.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://on-line-poker.allgames4u.net/roulette.html ROULETTE]
+
[http://on-line-poker.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://on-line-poker.allgames4u.net/slots-online.html SLOTS ONLINE]
+
[http://on-line-poker.allgames4u.net/video-poker.html VIDEO POKER]
+
[http://online-casino-gambling.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://online-casino-gambling.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://online-casino-gambling.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://online-casino-gambling.allgames4u.net/blackjack.html BLACKJACK]
+
[http://online-casino-gambling.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://online-casino-gambling.allgames4u.net/casinos.html CASINOS]
+
[http://online-casino-gambling.allgames4u.net/index.html ONLINE-CASINO-GAMBLING]
+
[http://online-casino-gambling.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://online-casino-gambling.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://online-casino-gambling.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://online-casino-gambling.allgames4u.net/slots-online.html SLOTS ONLINE]
+
[http://online-casino-gambling.allgames4u.net/video-poker.html VIDEO POKER]
+
[http://online-casino-games.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://online-casino-games.allgames4u.net/best-casino.html BEST CASINO]
+
[http://online-casino-games.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://online-casino-games.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://online-casino-games.allgames4u.net/best-poker.html BEST POKER]
+
[http://online-casino-games.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://online-casino-games.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://online-casino-games.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://online-casino-games.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://online-casino-games.allgames4u.net/blackjack.html BLACKJACK]
+
[http://online-casino-games.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://online-casino-games.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://online-casino-games.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://online-casino-games.allgames4u.net/craps-online.html CRAPS ONLINE]
+
[http://online-casino-games.allgames4u.net/craps.html CRAPS]
+
[http://online-casino-games.allgames4u.net/free-casino.html FREE CASINO]
+
[http://online-casino-games.allgames4u.net/index.html ONLINE-CASINO-GAMES]
+
[http://online-casino-games.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://online-casino-games.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://online-casino-games.allgames4u.net/internet-casinos.html INTERNET CASINOS]
+
[http://online-casino-games.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://online-casino-games.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://online-casino-games.allgames4u.net/internet-slots.html INTERNET SLOTS]
+
[http://online-casino-games.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://online-casino-games.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://online-casino-games.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://online-casino-games.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://online-casino-games.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://online-casino-games.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://online-casino-games.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://online-casino-games.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://online-casino-games.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://online-casino-games.allgames4u.net/play-poker.html PLAY POKER]
+
[http://online-casino-games.allgames4u.net/play-roulette.html PLAY ROULETTE]
+
[http://online-casino-games.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://online-casino-games.allgames4u.net/poker-online.html POKER ONLINE]
+
[http://online-casino-games.allgames4u.net/poker-room.html POKER ROOM]
+
[http://online-casino-games.allgames4u.net/poker.html POKER]
+
[http://online-casino-games.allgames4u.net/roulette-game.html ROULETTE GAME]
+
[http://online-casino-games.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://online-casino-games.allgames4u.net/slots.html SLOTS]
+
[http://online-casino-news.allgames4u.net/best-casino.html BEST CASINO]
+
[http://online-casino-news.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://online-casino-news.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://online-casino-news.allgames4u.net/best-poker.html BEST POKER]
+
[http://online-casino-news.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://online-casino-news.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://online-casino-news.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://online-casino-news.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://online-casino-news.allgames4u.net/casino.html CASINO]
+
[http://online-casino-news.allgames4u.net/casinos.html CASINOS]
+
[http://online-casino-news.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://online-casino-news.allgames4u.net/craps-online.html CRAPS ONLINE]
+
[http://online-casino-news.allgames4u.net/craps.html CRAPS]
+
[http://online-casino-news.allgames4u.net/free-casino.html FREE CASINO]
+
[http://online-casino-news.allgames4u.net/free-casinos.html FREE CASINOS]
+
[http://online-casino-news.allgames4u.net/index.html ONLINE-CASINO-NEWS]
+
[http://online-casino-news.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://online-casino-news.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://online-casino-news.allgames4u.net/internet-casinos.html INTERNET CASINOS]
+
[http://online-casino-news.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://online-casino-news.allgames4u.net/internet-slots.html INTERNET SLOTS]
+
[http://online-casino-news.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://online-casino-news.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://online-casino-news.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://online-casino-news.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://online-casino-news.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://online-casino-news.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://online-casino-news.allgames4u.net/play-casinos.html PLAY CASINOS]
+
[http://online-casino-news.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://online-casino-news.allgames4u.net/play-poker.html PLAY POKER]
+
[http://online-casino-news.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://online-casino-news.allgames4u.net/poker.html POKER]
+
[http://online-casino-news.allgames4u.net/slots-online.html SLOTS ONLINE]
+
[http://online-casino-news.allgames4u.net/video-poker.html VIDEO POKER]
+
[http://online-casino.allgames4u.net/best-casino.html BEST CASINO]
+
[http://online-casino.allgames4u.net/best-poker.html BEST POKER]
+
[http://online-casino.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://online-casino.allgames4u.net/casinos.html CASINOS]
+
[http://online-casino.allgames4u.net/craps-online.html CRAPS ONLINE]
+
[http://online-casino.allgames4u.net/craps.html CRAPS]
+
[http://online-casino.allgames4u.net/index.html ONLINE-CASINO]
+
[http://online-casino.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://online-casino.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://online-casino.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://online-casino.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://online-casino.allgames4u.net/play-casinos.html PLAY CASINOS]
+
[http://online-casino.allgames4u.net/poker-online.html POKER ONLINE]
+
[http://online-casino.allgames4u.net/poker-room.html POKER ROOM]
+
[http://online-casino.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://online-casino.allgames4u.net/slots-online.html SLOTS ONLINE]
+
[http://online-casinos.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://online-casinos.allgames4u.net/best-casino.html BEST CASINO]
+
[http://online-casinos.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://online-casinos.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://online-casinos.allgames4u.net/best-poker.html BEST POKER]
+
[http://online-casinos.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://online-casinos.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://online-casinos.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://online-casinos.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://online-casinos.allgames4u.net/blackjack.html BLACKJACK]
+
[http://online-casinos.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://online-casinos.allgames4u.net/casino.html CASINO]
+
[http://online-casinos.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://online-casinos.allgames4u.net/casinos.html CASINOS]
+
[http://online-casinos.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://online-casinos.allgames4u.net/craps-online.html CRAPS ONLINE]
+
[http://online-casinos.allgames4u.net/craps.html CRAPS]
+
[http://online-casinos.allgames4u.net/free-casino.html FREE CASINO]
+
[http://online-casinos.allgames4u.net/index.html ONLINE-CASINOS]
+
[http://online-casinos.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://online-casinos.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://online-casinos.allgames4u.net/internet-casinos.html INTERNET CASINOS]
+
[http://online-casinos.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://online-casinos.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://online-casinos.allgames4u.net/internet-roulette.html INTERNET ROULETTE]
+
[http://online-casinos.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://online-casinos.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://online-casinos.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://online-casinos.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://online-casinos.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://online-casinos.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://online-casinos.allgames4u.net/play-poker.html PLAY POKER]
+
[http://online-casinos.allgames4u.net/play-roulette.html PLAY ROULETTE]
+
[http://online-casinos.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://online-casinos.allgames4u.net/poker-game.html POKER GAME]
+
[http://online-casinos.allgames4u.net/poker-room.html POKER ROOM]
+
[http://online-casinos.allgames4u.net/poker.html POKER]
+
[http://online-casinos.allgames4u.net/roulette-game.html ROULETTE GAME]
+
[http://online-casinos.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://online-casinos.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://online-casinos.allgames4u.net/video-poker.html VIDEO POKER]
+
[http://party-poker.allgames4u.net/best-poker.html BEST POKER]
+
[http://party-poker.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://party-poker.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://party-poker.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://party-poker.allgames4u.net/casino.html CASINO]
+
[http://party-poker.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://party-poker.allgames4u.net/casinos.html CASINOS]
+
[http://party-poker.allgames4u.net/craps.html CRAPS]
+
[http://party-poker.allgames4u.net/index.html PARTY-POKER]
+
[http://party-poker.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://party-poker.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://party-poker.allgames4u.net/internet-roulette.html INTERNET ROULETTE]
+
[http://party-poker.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://party-poker.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://party-poker.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://party-poker.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://party-poker.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://party-poker.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://party-poker.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://party-poker.allgames4u.net/play-poker.html PLAY POKER]
+
[http://party-poker.allgames4u.net/poker-game.html POKER GAME]
+
[http://party-poker.allgames4u.net/poker-online.html POKER ONLINE]
+
[http://party-poker.allgames4u.net/poker-room.html POKER ROOM]
+
[http://party-poker.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://poker-tables.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://poker-tables.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://poker-tables.allgames4u.net/best-poker.html BEST POKER]
+
[http://poker-tables.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://poker-tables.allgames4u.net/blackjack.html BLACKJACK]
+
[http://poker-tables.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://poker-tables.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://poker-tables.allgames4u.net/casino.html CASINO]
+
[http://poker-tables.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://poker-tables.allgames4u.net/casinos.html CASINOS]
+
[http://poker-tables.allgames4u.net/craps.html CRAPS]
+
[http://poker-tables.allgames4u.net/free-casinos.html FREE CASINOS]
+
[http://poker-tables.allgames4u.net/index.html POKER-TABLES]
+
[http://poker-tables.allgames4u.net/internet-casinos.html INTERNET CASINOS]
+
[http://poker-tables.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://poker-tables.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://poker-tables.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://poker-tables.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://poker-tables.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://poker-tables.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://poker-tables.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://poker-tables.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://poker-tables.allgames4u.net/play-casinos.html PLAY CASINOS]
+
[http://poker-tables.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://poker-tables.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://poker-tables.allgames4u.net/poker.html POKER]
+
[http://poker-tables.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://poker-tables.allgames4u.net/slots.html SLOTS]
+
[http://poker-tables.allgames4u.net/video-poker.html VIDEO POKER]
+
[http://poker.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://poker.allgames4u.net/best-casino.html BEST CASINO]
+
[http://poker.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://poker.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://poker.allgames4u.net/best-poker.html BEST POKER]
+
[http://poker.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://poker.allgames4u.net/blackjack.html BLACKJACK]
+
[http://poker.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://poker.allgames4u.net/casinos.html CASINOS]
+
[http://poker.allgames4u.net/craps.html CRAPS]
+
[http://poker.allgames4u.net/index.html POKER]
+
[http://poker.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://poker.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://poker.allgames4u.net/internet-casinos.html INTERNET CASINOS]
+
[http://poker.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://poker.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://poker.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://poker.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://poker.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://poker.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://poker.allgames4u.net/play-casinos.html PLAY CASINOS]
+
[http://poker.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://poker.allgames4u.net/play-roulette.html PLAY ROULETTE]
+
[http://poker.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://poker.allgames4u.net/poker-game.html POKER GAME]
+
[http://poker.allgames4u.net/poker-room.html POKER ROOM]
+
[http://poker.allgames4u.net/roulette.html ROULETTE]
+
[http://poker.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://poker.allgames4u.net/slots.html SLOTS]
+
[http://roulette.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://roulette.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://roulette.allgames4u.net/blackjack-game.html BLACKJACK GAME]
+
[http://roulette.allgames4u.net/index.html ROULETTE]
+
[http://roulette.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://roulette.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://roulette.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://roulette.allgames4u.net/play-roulette.html PLAY ROULETTE]
+
[http://roulette.allgames4u.net/poker-game.html POKER GAME]
+
[http://roulette.allgames4u.net/poker-room.html POKER ROOM]
+
[http://roulette.allgames4u.net/roulette-game.html ROULETTE GAME]
+
[http://roulette.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://roulette.allgames4u.net/roulette.html ROULETTE]
+
[http://rules-poker.allgames4u.net/best-blackjack.html BEST BLACKJACK]
+
[http://rules-poker.allgames4u.net/best-casino.html BEST CASINO]
+
[http://rules-poker.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://rules-poker.allgames4u.net/best-craps.html BEST CRAPS]
+
[http://rules-poker.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://rules-poker.allgames4u.net/casino-games.html CASINO GAMES]
+
[http://rules-poker.allgames4u.net/casino-online.html CASINO ONLINE]
+
[http://rules-poker.allgames4u.net/casinos.html CASINOS]
+
[http://rules-poker.allgames4u.net/craps-game.html CRAPS GAME]
+
[http://rules-poker.allgames4u.net/free-casino.html FREE CASINO]
+
[http://rules-poker.allgames4u.net/index.html RULES-POKER]
+
[http://rules-poker.allgames4u.net/internet-blackjack.html INTERNET BLACKJACK]
+
[http://rules-poker.allgames4u.net/internet-craps.html INTERNET CRAPS]
+
[http://rules-poker.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://rules-poker.allgames4u.net/internet-roulette.html INTERNET ROULETTE]
+
[http://rules-poker.allgames4u.net/internet-slots.html INTERNET SLOTS]
+
[http://rules-poker.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://rules-poker.allgames4u.net/online-casinos.html ONLINE CASINOS]
+
[http://rules-poker.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://rules-poker.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://rules-poker.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://rules-poker.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://rules-poker.allgames4u.net/play-craps.html PLAY CRAPS]
+
[http://rules-poker.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://rules-poker.allgames4u.net/poker-game.html POKER GAME]
+
[http://rules-poker.allgames4u.net/poker-room.html POKER ROOM]
+
[http://rules-poker.allgames4u.net/poker.html POKER]
+
[http://rules-poker.allgames4u.net/roulette.html ROULETTE]
+
[http://rules-poker.allgames4u.net/slots-game.html SLOTS GAME]
+
[http://sports-gambling.allgames4u.net/best-casino.html BEST CASINO]
+
[http://sports-gambling.allgames4u.net/best-casinos.html BEST CASINOS]
+
[http://sports-gambling.allgames4u.net/best-roulette.html BEST ROULETTE]
+
[http://sports-gambling.allgames4u.net/best-slots.html BEST SLOTS]
+
[http://sports-gambling.allgames4u.net/blackjack-online.html BLACKJACK ONLINE]
+
[http://sports-gambling.allgames4u.net/casino.html CASINO]
+
[http://sports-gambling.allgames4u.net/casinos-online.html CASINOS ONLINE]
+
[http://sports-gambling.allgames4u.net/free-casino.html FREE CASINO]
+
[http://sports-gambling.allgames4u.net/free-casinos.html FREE CASINOS]
+
[http://sports-gambling.allgames4u.net/index.html SPORTS-GAMBLING]
+
[http://sports-gambling.allgames4u.net/internet-casino.html INTERNET CASINO]
+
[http://sports-gambling.allgames4u.net/internet-casinos.html INTERNET CASINOS]
+
[http://sports-gambling.allgames4u.net/internet-poker.html INTERNET POKER]
+
[http://sports-gambling.allgames4u.net/internet-roulette.html INTERNET ROULETTE]
+
[http://sports-gambling.allgames4u.net/online-blackjack.html ONLINE BLACKJACK]
+
[http://sports-gambling.allgames4u.net/online-casino.html ONLINE CASINO]
+
[http://sports-gambling.allgames4u.net/online-craps.html ONLINE CRAPS]
+
[http://sports-gambling.allgames4u.net/online-poker.html ONLINE POKER]
+
[http://sports-gambling.allgames4u.net/online-roulette.html ONLINE ROULETTE]
+
[http://sports-gambling.allgames4u.net/online-slots.html ONLINE SLOTS]
+
[http://sports-gambling.allgames4u.net/play-blackjack.html PLAY BLACKJACK]
+
[http://sports-gambling.allgames4u.net/play-casino.html PLAY CASINO]
+
[http://sports-gambling.allgames4u.net/play-casinos.html PLAY CASINOS]
+
[http://sports-gambling.allgames4u.net/play-poker.html PLAY POKER]
+
[http://sports-gambling.allgames4u.net/play-slots.html PLAY SLOTS]
+
[http://sports-gambling.allgames4u.net/poker-game.html POKER GAME]
+
[http://sports-gambling.allgames4u.net/poker-online.html POKER ONLINE]
+
[http://sports-gambling.allgames4u.net/poker-room.html POKER ROOM]
+
[http://sports-gambling.allgames4u.net/poker.html POKER]
+
[http://sports-gambling.allgames4u.net/roulette-game.html ROULETTE GAME]
+
[http://sports-gambling.allgames4u.net/roulette-online.html ROULETTE ONLINE]
+
[http://sports-gambling.allgames4u.net/slots.html SLOTS]</div>
+

Revision as of 08:34, 11 July 2018

Documentation linux.png This section assumes use of gdb, from the console. There are also specific notes on Windows Debugging or graphical tools on Mac OS X and hints on Build Problems Debugging.

Building with debugging symbols

OO.o includes a way to add debugging code in per module, via the build debug=true command in each module. This also adds lots of runtime assertions, churning warnings etc. in addition to debug symbols - which can be useful. To do just a plain build with debug symbols though use build debug=true dbg_build_only=true or in later versions use build debug=true dbglevel=2 for max output and dbglevel=1 or 0 for less output.

You can also configure OO.o with --enable-symbols to build with symbolic generation.

gdb invocation

If you debug with gdb, you may find that execution stops due to signals at inappropriate locations, especially if running against libgcj and need to debug ignoring its garbage-collection. Best invocation is...

gdb ./soffice.bin
(gdb) handle SIGPWR nostop noprint
(gdb) handle SIGXCPU nostop noprint
(gdb) handle SIG33 nostop noprint
(gdb) run -norestore -writer

replace -writer with -draw/-impress/-calc/... as appropriate. The -norestore option prevents display of the crash reporter (as one frequently kills office during debugging).

The recommended .gdbinit file

You can add the handle commands from above to your ~/.gdbinit to save some typing. Also you can define there macros to print the content of strings even in cases where dbg_dump() does not work. The following is such recommended ~/.gdbinit file:

set history filename ~/.gdbhistory
set history save on
 
handle SIGPWR nostop noprint
handle SIGXCPU nostop noprint
handle SIG33 nostop noprint
 
tabset 4
 
# define "pu" command to display sal_Unicode *
define pu
  set $uni = $arg0
  set $len = $arg1
  set $i = 0
  printf "\""
  while (*$uni && $i++<$len && $i<255)
    if (*$uni < 0x80)
      printf "%c", *(char*)$uni++
    else
      printf "\\x%x", *(short*)$uni++
    end
  end
  printf "\"\n"
end
 
# define "pus" command to display rtl_uString
define pus
  if ($arg0.buffer)
    pu $arg0.buffer $arg0.length
  else
    print "Invalid/non-initialized rtl_uString."
  end
end
 
# define "pou" command to display rtl::OUString
define pou
  if ($arg0.pData)
    pus $arg0.pData
  else
    print "Invalid/non-initialized OUString."
  end
end
 
# define "ptu" command to display tools (Uni)String
define ptu
  if ($arg0.mpData)
    pu $arg0.mpData->maStr $arg0.mpData->mnLen
  else
    print "Invalid/non-initialized tools String."
  end
end
 
# define "ps" command that will autodetect the type of the string,
# and call a function accordingly
define ps
  if ($arg0.pData)
    pou $arg0
  else
    if ($arg0.mpData)
      ptu $arg0
    else
      if ($arg0.buffer)
        pus $arg0
      else
        set $len = $arg1
        if ($len)
          pu $arg0 $len
        else
          # first 20 (unicode) chars
          pu $arg0 20
        end
      end
    end
  end
end
document ps
ps somestring [len]

Print the content of pretty much any OpenOffice.org string. The length parameter is only used for sal_Unicode* strings: if not provided, the first 20 characters will be printed for those kinds of strings. end </pre>

With this, you can use pou the_OUString in gdb to print OUString named the_OUString, similarly pus the_rtl_uString for rtl_uString and pu the_p_sal_Unicode for sal_Unicode *. For tools's UniString (or just String for historical reasons) you can use ptu the_String

ps anyString autodetects anyString and calls one of the p* macros accordingly.

Some further options

Here are some other options some people might find helpful:

  • set pagination off: When displaying a backtrace don't stop at each page of output. The user can scroll up the buffer with Shift+PgUp/PgDn.
  • set history expansion on: Enable tab expansion of already entered commands (history)

Starting at the beginning

We start in 'main' with a sal wrapper, that calls vcl/source/app/svmain.cxx (SVMain). It invokes Main on pSVData->mpApp; but pSVData is an in-line local. To debug this use the pImplSVData global variable. eg:

     p pImplSVData->maAppData

This 'Main' method is typically: desktop/source/app/app.cxx (Main).

Examining strings

We have already seen that OO.o has its own set of string classes, none of which gdb understands. You need to use: (gdb) print dbg_dump(sWhatEver) to print the contents of a UniString/ByteString/rtl::OUString/rtl::OString regardless of the type when debugging C++ code. See Caolan's write-up for details.

the functions dbg_dump() may not be available in gdb due to link. Just copy them in your current source file from '/sal/rtl/source/debug_print' and add the associated includes #include <rtl/strbuf.hxx> #include <rtl/ustring.hxx>. gdb should recognize them now.

Another way is to use the macros from the recommended .gdbinit file. Unfortunately, so far works only for OUString, sal_Unicode *, and rtl_uString.

Getting the build order right

The build dependencies of the modules are clearly crucial to getting a clean build. When you type 'build' in a module, first build examines prj/build.list, eg.neon/prj/build.lst:

       xh      neon  :  soltools external expat NULL

this specifies that 'soltools', 'external' and 'expat' have to be satisfactorily built and delivered before neon can be built. Occasionally these rules get broken, and people don't notice for a while.

It crashes, but only in gdb

What fun — you symlinked desktop/unxlngi4.pro/bin/soffice to soffice.bin in your install tree didn't you. That works fine if you just run it, but it seems gdb unpacks the symlink and passes a fully qualified path as argv[0], which defeats the hunting for the binary in the path, so it assigns the program base path as /opt/OpenOffice/OOO_STABLE_1/desktop/unxlngi4.pro/bin and starts looking for (eg. applicat.rdb) in there. Of course when it fails to find any setup information, it silently crashes somewhere else yards away from the original problem.

It crashes, but doesn't crash

For various reasons signal handlers are trapped and life can get rather confusing; thus it's best for builders to apply something like this:

--- sal/osl/unx/signal.c
+++ sal/osl/unx/signal.c
@@ -188,6 +188,8 @@ static sal_Bool InitSignal()
             bSetILLHandler = sal_True;
        }
 
+       bSetSEGVHandler = bSetWINCHHandler = bSetILLHandler = bDoHardKill = sal_False;
+
        SignalListMutex = osl_createMutex();
 
        act.sa_handler = SignalHandlerFunction;

I can't find the code from the trace

Some methods, are described as having a special linkage, such that they can be used in callbacks; these typically have a prefix: 'LinkStub', so search for the latter part of the identifier in a freetext search. eg.

      IMPL_LINK( Window, ImplHandlePaintHdl, void*, EMPTYARG )

builds the 'LinkStubImplHandlePaintHdl' method.

How can I re-build just the files I see in the trace

Often when you run gdb on a build without debugging symbols, you get an unhelpful gdb trace, but yet you can't afford the time/space to recompile all of OO.o with debugging symbols. Thus we have created a small perl helper, which will hunt for and touch files containing the symbols from your trace. This sub-set can then be re-built with debugging enabled for a better trace next time around:

    gdb ./soffice.bin
    ...
    bt
#0  0x40b4e0a1 in kill () from /lib/libc.so.6
#1  0x409acfe6 in raise () from /lib/libpthread.so.0
#2  0x447bcdbd in SfxMedium::DownLoad(Link const&) () from ./libsfx641li.so
#3  0x447be151 in SfxMedium::SfxMedium(String const&, unsigned short, unsigned char, SfxFilter const*, SfxItemSet*) ()
   from ./libsfx641li.so
#4  0x448339d3 in getCppuType(com::sun::star::uno::Reference<com::sun::star::document::XImporter> const*) () from ./libsfx641li.so
...
    quit
    cd base/OOO_STABLE_1/sfx2
    ootouch SfxMedium
    build debug=true
    

Thus, all files referencing or implementing anything with SfxMedium will be touched, and hence rebuilt with debugging symbols.

ootouch is not available upstream: it is available through ooo-build.

How can I re-build all the files in one source directory

If you want to recompile the code in just your current directory, you can use the killobj dmake target to remove the object files:

    dmake killobj
    dmake
    

It always crashes in sal_XErrorHdl

You are a victim of asynchronous X error reporting; export SAL_SYNCHRONIZE=1 will make all the X traffic synchronous, and report the error by the method that caused it, it'll also make OO.o far slower, and the timing different.

It silently fails to load my word file

Caolan suggests: put breakpoints in ww8par.cxx top and tail of SwWW8ImplReader::LoadDoc, and confirm that the document gets as far as the import filter.

A handy human place to put a breakpoint is in SwWW8ImplReader::ReadPlainChars, you can see chunks of text as they are read in. Alternatively SwWW8ImplReader::AppendTxtNode as each paragraph is inserted.

How do I use the debug console ?

OpenOffice.org contains some debugging infrastructure; enabling it is pretty easy - what you need is a so-called Non-Product Build, see details and debug settings dialog there.

Note that libraries from product and non-product builds are usually incompatible, so don't mix them in the same installation.

To actually fire up the debug settings dialog, press <alt><shift><ctrl>-D.

Draw/Impress text edit debugging

When running a Non-Product Build, live edit mode in Draw/Impress text boxes has an extra debug hotkey: pressing <ctrl><alt>-F2 writes information about the currently edited text into a debug.log file (currently Windows-only).

Excel Interop debugging

To dump the contents of binary MS Excel files while loading them into OpenOffice.org, first get a local copy of the code module "oox". Then:

  • XLS files (BIFF2-BIFF8): define the environment variable OOO_BIFFDUMPER pointing to file://..../oox/source/dump/biffdumper.ini (full path needed). Then run soffice.bin foo.xls and you should get a foo.xls.dump directory in the same directory with the debug data in it. The new directory will contain the complete storage structure with all original streams, and the decoded streams with ".dump" suffix.
  • XLSX, XLSM, XLSB files: define the environment variable OOO_XLSBDUMPER pointing to file://..../oox/source/dump/xlsbdumper.ini (full path needed). Then run soffice.bin foo.xlsb. XML streams will be pretty-printed, and known binary streams (e.g. the VBA project, and the streams used in XLSB format) are decoded.
  • also PPTX, PPTM files are supported: define the environment variable OOO_PPTXDUMPER pointing to file://..../oox/source/dump/pptxdumper.ini (full path needed). Then run soffice.bin foo.pptx.

Note: this requires a debug build of the oox module. To easily get such a build, execute the following within your sc directory: build -- killobj ; build debug=true

The trace shows a crash in 'poll'

OO.o is a fairly threaded program, you're probably just looking at the wrong thread: there are not likely to be bugs in poll. Use thread apply all backtrace to get a backtrace of all threads - this will most likely fail. When it does do: thread 1 then bt - most crashers occur in the 'main' thread.

What does this trace mean ?

There are several typical stack-traces that come up again and again, one would be:

#15 0x4164a501 in raise () from /lib/tls/libc.so.6
#16 0x4164bcd9 in abort () from /lib/tls/libc.so.6
#17 0x415fb5a5 in std::set_unexpected ()
   from /home/mnagashree/m72install/program/libstdc++.so.5
#18 0x415fb5e2 in std::terminate ()
   from /home/mnagashree/m72install/program/libstdc++.so.5
#19 0x415fb69c in __cxa_rethrow ()
    

This section of trace means (essentially) that an exception was thrown - but there was no-one trying to catch it. Often this means there was a missing 'try {} catch()' clause in one of the calling frames.

A great way to debug exceptions is to add a breakpoint in catch/throw, do this with catch throw or catch catch in gdb.

Useful places to put breakpoints

If you have compiled with debugging enabled: build debug=true it is possible that you get some nice churning debug / assertion failure - and you want to get a pleasant & detailed stack-trace: to do that do break osl_assertFailedLine.

STLport and checking iterators

The STL is a powerful tool but it also makes it easy - in the grand old C/C++ tradition - to shoot one selves in the foot, as we all know. STL containers and algorithms are now pervasive in OOo, so there is a need to validate the use of STL constructs in OO.o to find hidden problems.

Fortunately the STLport library - the default STL implementation for OO.o - has a powerful debug mode, and it's easy to use. Since SRC680 m128 it is possible to use the environment variable USE_STLP_DEBUG to switch on the STLport debug mode, since SRC680 m150 it works for Windows, too

The most useful part of the STLport debug mode is iterator checking. Doing the OO.o smoke test and some little additional random testing we already found a number of questionable STL constructs.

Only code paths which are exercised will be tested by the STLport debug mode, though. If STLport finds a questionable STL usage it will throw an assertion and terminate. It is usually quite easy to extract a precise stack trace.

Some notes:

  • STLport debug mode iterators are no pointers! We've cleaned up all occurrences of the lazy - and wrong - usages of iterators as pointers in SRC680 m128/m150, but maybe something new has already crept in. This clean up also helps with other STL implementations, like the one which comes with gcc-4.x
  • A complete recompile is necessary, the debug modes renders all objects with STL constructs binary incompatible
  • The STLport debug mode breaks the complexity assertions of the STL. Theoretically some operations should be much slower in debug mode than in product mode. In practice I didn't notice a real slowdown of OO.o.
Personal tools