Monday, January 03, 2011

How Oracle Locking Works

Myth: Oracle has a Central Locking System

A week or so ago I started a new series – 100 Things You Probably didn’t Know about Oracle Database . Part 1 of the series proved quite popular, as I surmise from the emails I received. Here is the Part 2 of the series.

When a transaction updates a row, it puts a lock so that no one can update the same row until it commits. When another transaction issues an update to the same row, it waits until the first one either commits or rolls back. After the first transaction performs a commit or rollback, the update by the second transaction is executed immediately, since the lock placed by the first transaction is now gone. How exactly does this locking mechanism work? Several questions come to mind in this context:


  1. Is there some kind of logical or physical structure called lock?
  2. How does the second transaction know when the first transaction has lifted the lock?
  3. Is there some kind of “pool” of such locks where transactions line up to get one?
  4. If so, do they line up to return it when they are done with the locking?
  5. Is there a maximum number of possible locks?
  6. Is there something called a block level lock? Since Oracle stores the rows in blocks, when all or the majority of rows in the blocks are locked by a single transaction, doesn’t it make sense for to lock the entire block to conserve the number of locks?
  7. The previous question brings up another question – does the number of active locks in the database at any point really matter?


If you are interested to learn about all this, please read on.

Lock Manager

Since locks convey information on who has what rows modified but not committed, anyone interested in making the update much check with some sort of system that is available across the entire database. So, it makes perfect sense to have a central locking system in the database, doesn’t it? But, when you think about it, a central lock manager can quickly become a single point of contention in a busy system where a lot of updates occur. Also, when a large number of rows are updated in a single transaction, an equally large number of locks will be required as well. The question is: how many? One can guess; but it will be at best a wild one. What if you guessed on the low side and the supply of available locks is depleted? In that case some transactions can’t get locks and therefore will have to wait (or, worse, abort). Not a pleasant thought in a system that needs to be scalable. To counter such a travesty you may want to make the available supply of locks really high. What is the downside of that action? Since each lock would potentially consume some memory, and memory is finite, it would not be advisable to create an infinite supply of locks.

Some databases actually have a lock manager with a finite supply of such locks. Each transaction must ask to get a lock from it before beginning and relinquish locks to it at the completion. In those technologies, the scalability of application suffers immensely as a result of the lock manager being the point of contention. In addition, since the supply of locks is limited, the developers need to commit frequently to release the locks for other transactions. When a large number of rows have locks on them, the database replaces the row locks with a block level lock to cover all the rows in the block – a concept known as lock escalation. Oracle does not follow that approach. In Oracle, there no central lock manager, no finite limit on locks and there is no such concept called lock escalation. The developers commit only when there is a logical need to do so; not otherwise.

Lock Management in Oracle

So, how is that approach different in case of Oracle? For starters, there is no central lock manager. But the information on locking has to be recorded somewhere. Where then? Well, consider this: when a row is locked, it must be available to the session, which means the session’s server process must have already accessed and placed the block in the buffer cache prior to the transaction occurring. Therefore, what is a better place for putting this information than right there in the block (actually the buffer in the buffer cache) itself?

Oracle does precisely that – it records the information in the block. When a row is locked by a transaction, that nugget of information is placed in the header of the block where the row is located. When another transaction wishes to acquire the lock on the same row, it has to access the block containing the row anyway (as you learned in Part 1 of this series) and upon reaching the block, it can easily confirm that the row is locked from the block header. A transaction looking to update a row in a different block puts that information on the header of that block. There is no need to queue behind some single central resource like a lock manager. Since lock information is spread over multiple blocks instead of a single place, this mechanism makes transactions immensely scalable.

Being the smart reader you are, you are now hopefully excited to learn more or perhaps you are skeptical. You want to know the nuts and bolts of this whole mechanism and, more, you want proof. We will see all that in a moment.

Transaction Address

Before understanding the locks, you should understand clearly what a transaction is and how it is addressed. A transaction starts when an update to data such as insert, update or delete occurs (or the intention to do so, e.g. SELECT FOR UPDATE) and ends when the session issues a commit or rollback. Like everything else, a specific transaction should have a name or an identifier to differentiate it from another one of the same type. Each transaction is given a transaction ID. When a transaction updates a row (it could also insert a new row or delete an existing one; but we will cover that little later in this article), it records two things:


  • The new value
  • The old value


The old value is recorded in the undo segments while the new value is immediately updated in the buffer where the row is stored. The data buffer containing the row is updated regardless of whether the transaction is committed or not. Yes, let me repeat – the data buffer is updated as soon as the transaction modifies the row (before commit). If you didn’t know that, please see the Part 1 of this series.

Undo information is recorded in a circular fashion. When new undo is created, it is stored in the next available undo “slot”. Each transaction occupies a record in the slot. After all the slots are exhausted and a new transaction arrives, the next processing depends on the state of the transactions. If the oldest transaction occupying any of the other slots is no longer active (that is either committed or rolled back), Oracle will reuse that slot. If none of the transactions is inactive, Oracle will have to expand the undo tablespace to make room. In the former case (where a transaction is no longer active and its information in undo has been erased by a new transaction), if a long running query that started before the transaction occurred selects the value, it will get an ORA-1555 error. But that will be covered in a different article in the future. If the tablespace containing the undo segment can’t extend due to some reason (such as in case of the filesystem being completely full), the transaction will fail.

Speaking of transaction identifiers, it is in the form of three numbers separated by periods. These three numbers are:


  • Undo Segment Number where the transaction records its undo entry
  • Slot# in the undo segment
  • Sequence# (or wrap) in the undo slot


This is sort of like the social security number of the transaction. This information is recorded in the block header. Let’s see the proof now through a demo.

Demo

First, create a table:

SQL> create table itltest (col1 number, col2 char(8));

Insert some rows into the table.

SQL> begin
  2     for i in 1..10000 loop 
  3             insert into itltest values (i,'x');
  4     end loop;
  5     commit;
  6  end;
  7  /

Remember, this is a single transaction. It started at the “BEGIN” line and ended at “COMMIT”. The 10,000 rows were all inserted as a part of the same transaction. To know the transaction ID of this transaction, Oracle provides a special package - dbms_transaction. Here is how you use it. Remember, you must use it in the same transaction. Let’s see:

SQL> select dbms_transaction.local_transaction_id from dual;

LOCAL_TRANSACTION_ID
------------------------------------------------------------------------


1 row selected.

Wait? There is nothing. The transaction ID returned is null. How come?

If you followed the previous section closely, you will realize that the transaction ends when a commit or rollback is issued. The commit was issued inside the PL/SQL block. So, the transaction had ended before you called the dbms_transaction is package. Since there was no transaction, the package returned null.

Let’s see another demo. Update one row:

SQL> update itltest set col2 = 'y' where col1 = 1;

1 row updated.

In the same session, check the transaction ID:

SQL> select dbms_transaction.local_transaction_id from dual;

LOCAL_TRANSACTION_ID
-------------------------------------------------------------------------
3.23.40484

1 row selected.

There you see – the transaction ID. The three numbers separated by period signify undo segment number, slot# and record# respectively. Now perform a commit:

SQL> commit;

Commit complete.

Check the transaction ID again:

SQL> select dbms_transaction.local_transaction_id from dual;

LOCAL_TRANSACTION_ID
-------------------------------------------------------------------------


1 row selected.

The transaction is gone so the ID is null, as expected.

Since the call to the package must be in the same transaction (and therefore in the same session), how can you check the transaction in a different session? In real life you will be asked to check transaction in other sessions, typically application sessions. Let’s do a slightly different test. Update the row one more time and check the transaction:

SQL> update itltest set col2 = 'y' where col1 = 1;

1 row updated.

SQL> select dbms_transaction.local_transaction_id from dual;

LOCAL_TRANSACTION_ID
-----------------------------------------------------------------------
10.25.31749

1 row selected.

From a different session, check for active transactions. This information is available in the view V$TRANSACTION. There are several columns; but we will look at four of the most important ones:


  • ADDR – the address of the transaction, which is a raw value
  • XIDUSN – the undo segment number
  • XIDSLOT – the slot#
  • XIDSQN – the sequence# or record# inside the slot


SQL> select addr, xidusn, xidslot, xidsqn
  2  from v$transaction;

ADDR         XIDUSN    XIDSLOT     XIDSQN
-------- ---------- ---------- ----------
3F063C48         10         25      31749

Voila! You see the transaction id of the active transaction from a different session. Compare the above output to the one you got from the call to dbms_transaction package. You can see that the transaction identifier shows the same set of numbers.

Interested Transaction List

You must be eager to know about the section of the block header that contains information on locking and how it records it. It is a simple data structure called "Interested Transaction List" (ITL), a list that maintains information on transaction. The ITL contains several placeholders (or slots) for transactions. When a row in the block is locked for the first time, the transaction places a lock in one of the slots. In other words, the transaction makes it known that it is interested in some rows (hence the term "Interested Transaction List"). When a different transaction locks another set of rows in the same block, that information is stored in another slot and so on. When a transaction ends after a commit or a rollback, the locks are released and the slot which was used to mark the row locks in the block is now considered free (although it is not updated immediately - fact about which you will learn later in a different installment).

[Updated Jan 22, 2011] [Thank you, Randolph Geist (info@www.sqltools-plusplus.org) for pointing it out. I follow his blog http://oracle-randolf.blogspot.com/, which is a treasure trove of information.
The row also stores a bit that represents the whether it is locked or not.
[end of Update Jan 22, 2011]


ITLs in Action

Let's see how ITLs really work. Here is an empty block. The block header is the only occupant of the block.


This is how the block looks like after a single row has been inserted:


Note, the row was inserted from the bottom of the block. Now, a second row has been inserted:



A session comes in and updates the row Record1, i.e. it places a lock on the row, shown by the star symbol. The lock information is recorded in the ITL slot in the block header:


The session does not commit yet; so the lock is active. Now a second session - Session 2 - comes in and updates row Record2. It puts a lock on the record - as stored in the ITL slot.


I have used two different colors to show the locks (as shown by the star symbol) and the color of the ITL entry.

As you can clearly see, when a transaction wants to update a specific row, it doesn’t have to go anywhere but the block header itself to know if the row is locked or not. All it has to do is to check the ITL slots. However ITL alone does not show with 100% accuracy that row is locked (again, something I will explain in a different installment). The transaction must go to the undo segment to check if the transaction has been committed. How does it know which specifci part of the undo segment to go to? Well, it has the information in the ITL entry. If the row is indeed locked, the transaction must wait and retry. As soon as the previous transaction ends, the undo information is updated and the waiting transaction completes its operation.

So, there is in fact a queue for the locks, but it's at the block level, not at the level of the entire database or even the segment.

Demo

The proof is in the pudding. Let’s see all this through a demo. Now that you know the transaction entry, let’s see how it is stored in the block header. To do that, first, we need to know which blocks to look for. So, we should get the blocks numbers where the table is stored:

SQL> select file_id, relative_fno, extent_id, block_id, blocks
  2  from dba_extents
  3  where segment_name = 'ITLTEST';

   FILE_ID RELATIVE_FNO  EXTENT_ID   BLOCK_ID     BLOCKS
---------- ------------ ---------- ---------- ----------
         7            7          0       3576          8
         7            7          1       3968          8
         7            7          2       3976          8
         7            7          3       3984          8

To check inside the block, we need to “dump” the contents of the block to a tracefile so that we can read it. From a different session issue a checkpoint so that the buffer data is now written to the dis:

SQL> alter system checkpoint;

Now dump the data blocks 3576 through 3583.

SQL> alter system dump datafile 7 block min 3576 block max 3583;

System altered.

This will create a tracefile in the user dump destination directory. In case of Oracle 11g, the tracefile will be in the diag structure under /diag/rdbms///trace directory. It will be most likely the last tracefile generated. You can also get the precise name by getting the OS process ID of the session:


SQL> select p.spid
  2  from v$session s, v$process p
  3  where s.sid = (select sid from v$mystat where rownum < 2)
  4  and p.addr = s.paddr
  5  /

SPID
------------------------
9202

1 row selected.
Now look for a file named _ora_9202.trc. Open the file in vi and search for the phrase “Itl”. Here is an excerpt from the file:


Itl           Xid                  Uba          Flag  Lck        Scn/Fsc
0x01   0x000a.019.00007c05  0x00c00288.1607.0e  ----    1  fsc 0x0000.00000000
0x02   0x0003.017.00009e24  0x00c00862.190a.0f  C---    0  scn 0x0000.02234e2b

This is where the information on row locking is stored. Remember, the row locking information is known as Interested Transaction List (ITL) and each ITL is stored in a “slot”. Here it shows two slots, which is the default number. Look for the one where the “Lck” column shows a value. It shows “1”, meaning one of the rows in the blocks is locked by a transaction. But, which transaction? To get that answer, note the value under the “Xid” column. It shows the transaction ID - 0x000a.019.00007c05. These numbers are in hexadecimal (as indicated by the 0x at the beginning of the number). Using the scientific calculator in Windows, I converted the values to decimal as 10, 25 and 31749 respectively. Do they sound familiar? Of course they do; they are exactly as reported by both the record in v$transaction and the dbms_transaction.local_transaction_id function call.

This is how Oracle determines that there is a transaction has locked the row and correlates it to the various components in the other areas – mostly the undo segments to determne if it is active. Now that you know undo segments holds the transaction details, you may want to know more about the segment. Remember, the undo segment is just a segment, like any other table, indexes, etc. It resides in a tablespace, which is on some datafile. To find out the specifics of the segment, we will look into some more columns of the view V$TRANSACTION:


SQL> select addr, xidusn, xidslot, xidsqn, ubafil, ubablk, ubasqn, ubarec,
  2  status, start_time, start_scnb, start_scnw, ses_addr
  3  from v$transaction;

ADDR         XIDUSN    XIDSLOT     XIDSQN     UBAFIL     UBABLK     UBASQN
-------- ---------- ---------- ---------- ---------- ---------- ----------
    UBAREC STATUS           START_TIME           START_SCNB START_SCNW SES_ADDR
---------- ---------------- -------------------- ---------- ---------- --------
3F063C48         10         25      31749          3        648       5639
        14 ACTIVE           12/30/10 20:00:25      35868240          0 40A73784


1 row selected.
The columns with names starting with UBA show the undo block address information. Look at the above output. The UBAFIL shows the file#, which is “3” in this case. Checking for the file_id: 

SQL> select * from dba_data_files
  2> where file_id = 3;

FILE_NAME
-------------------------------------------------------------------------
   FILE_ID TABLESPACE_NAME                     BYTES     BLOCKS STATUS
---------- ------------------------------ ---------- ---------- ---------
RELATIVE_FNO AUT   MAXBYTES  MAXBLOCKS INCREMENT_BY USER_BYTES USER_BLOCKS
------------ --- ---------- ---------- ------------ ---------- -----------
ONLINE_
-------
+DATA/d112d2/datafile/undotbs1.260.722742813
         3 UNDOTBS1                       4037017600     492800 AVAILABLE
           3 YES 3.4360E+10    4194302          640 4035969024      492672
ONLINE


1 row selected.

Note the UBASQN (which is the undo block sequence#) value in the earlier output, which was 5639. Let’s revisit the ITL entries in the dump of block:
Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x000a.019.00007c05  0x00c00288.1607.0e  ----    1  fsc 0x0000.00000000
0x02   0x0003.017.00009e24  0x00c00862.190a.0f  C---    0  scn 0x0000.02234e2b

Look at the entry under the Uba column: 0x00c00288.1607.0e. As indicated by the “0x” at the beginning, these are in hexadecimal. Using a scientific calculator, let’s convert them. 1607 in hex means 5639 in decimal – the UBA Sequence# (UBASQN). The value “e” is 14 in decimal, which corresponds to the UBAREC. Finally the value 288 is 648 in decimal, which is the UBABLK. Now you see how the information is recorded in the block header and is also available to the DBA through the view V$TRANSACTION. 

Let’s see some more important columns of the view. A typical database will have many sessions; not just one. Each session may have an active transaction, which means you have to link sessions to transactions to generate meaningful information. The transaction information also contains the session link. Note the column SES_ADDR, which is the address of the session that issued the transaction. From that, you can get the session information



SQL> select sid, username
  2  from v$session
  3  where saddr = '40A73784';

SID USERNAME
--- --------
123 ARUP

There you go – you now have the SID of the session. And now that you know the SID, you can look up any other relevant data on the session from the view V$SESSION.

Takeaways

Here is a summary of what you learned so far:
  1. Transaction in Oracle starts with a data update (or intention to update) statement. Actually there are some exceptions which we will cover in a later article.
  2. It ends when a commit or rollback is issued
  3. A transaction is identified by a transaction ID (XID) which is a set of three numbers – undo segment#, undo slot# and undo record# - separated by periods.
  4. You can view the transaction ID in the session itself by calling dbms_transaction.local_transaction_id function.
  5. You can also check all the active transactions in the view v$transaction, where the columns XIDUSN, XIDSLOT and XIDSQN denote the undo segment#, undo slot# and undo rec# - the values that make up the transaction ID.
  6. The transaction information is also stored in the block header. You can check it by dumping the block and looking for the term “Itl”.
  7. The v$transaction view also contains the session address under SES_ADDR column, which can be used to join with the SADDR column of v$session view to get the session details.
  8. From the session details, you can find out other actions by the session such as the username, the SQL issues, the machine issued from, etc.

I hope you found it useful. As always, I will be grateful to know how you liked it. In the next installment we will examine how these ITLs could cause issues and how to identify or resolve them.


62 comments:

The Human Fly said...

Excellent piece of information about locking behavior in Oracle. Kudos for your efforts to write up such a nice and deep drive article. I knew how much time and efforts it will take before writing any article, but, we can't wait for the other installments from you Arup.

Andy Campbell said...

The explanation of ITLs is slightly wrong.

A new transaction trying to take a lock on a row looks at the ITL information, as you say, if it finds a record for the row its trying to lock it then looks at the undo segment referenced to ensure its an active transaction.

If there is an active transaction then it waits - monitoring the undo segment for the transaction to finish - not the ITL.

Once a transaction is finished it does not go back and update the ITL record. Commits are almost instantaneous, however much work is done - that wouldn't be the case if it had to go back and update every blocks ITL.

This also means that updated (dirty blocks) can be flushed out to disk and do not need to be re-read on commit;

Otherwise a good article.

Surachart Opun said...

Thank You for great article. It's useful for me :)

Unknown said...

Extraordinarily clear article! Extremely useful.

Arup Nanda said...

@The Human Fly,@Gus, @Surachart - thank you for the comments.

@James - I agree with your comment. I wanted to balance between the amount of information to present and the specific details so as not to confuse the readers. I am going explain that in more detail in the next installment on ITL, dirty block flushing; so I decided to defer it. But now that I see your comment, I agree that it may mmislead. so, I edited the content just about enough to make the point across without information overload. Hope that works. And, thhey will be explained well in the next installment.

Yogi said...

I think what james said, invites another article on "delayed block commit".

Good explanation, its always good to revisit what you know and firm it further.

-Yogi

Anonymous said...

Arup,

First of all thanks a lot for your article. It's very valuable. But I have one point that remains unclear for me. In the article you show how determine the session which locked a row in a block analyzing block dump, but I didn't understand how determine the row which is being locked.

Arup Nanda said...

@Yogi - that is coming in the next installment

Arup Nanda said...

@Panick hang on; it's coming on the next one.

Baskar said...

Hi Arup Sir,

Wish you a Happy and Prosperous New Year. Very Nice Article. Its just wonderful.

thanks,
baskar.l

Jared said...

Excellent article Arup, thanks for taking the time to write it.

Anonymous said...

Very good idea to put this stuff into a Blog posting! I have explained it so many times in my courses over the last decade that it seems obvious to me, but as with so many other things about Oracle Databases, it just ain't so for many others :-) Good one!

Alberto said...

Great article and good explanation, Arup, very easy to read, i'm agree to you for @James comment.
Thank to @James too for his good analysis.
I'll wait your next installment..
Ciao

Alberto

Anonymous said...

Hi Sir,

Nice information.Thanks for the article. :)

"Here it shows two slots (actually there is only one slot for a table, by default).".

I have seen that though the data dictionary view shows ini_trans as "1", actually its not so.In 10g, if we dump a block we can see 3 initrans (by default for data block)and for 9i and 11g i have 2, though the dictionary view says 1.

http://aprakash.wordpress.com/2010/12/18/oracle-tables-initrans/

Regards,
Anand

Srikar Dasari, Hyderabad said...

@Arup
Thanks for your great blog and information on ITl. That was very helpful and easily understandable.
You have shared the diagrams as well, that was more appreciable.
Waiting for new installments :)

Arup Nanda said...

@Anand - I sincerely apologize for the delay. Yes, you are right; the default is 2.

JAMSHER KHAN said...
This comment has been removed by the author.
JAMSHER KHAN said...

Wonderful job

Sree said...

HI Arup,

Good day to you!.

Under the session Heading "Transaction Address" , i see - "If none of the transactions is active, Oracle will have to expand the undo tablespace to make room"

My understanding is "if all of the transaction are active, Oracle will have to expand the undo tablespace to make room"

NONE should be replace by ALL Kindly correct my understanding if am wrong


Thanks & regards,

Sree

Arup Nanda said...

@Sree - you are right. That was a typo and I have corrected it. It should have read "If none of the transactions is inactive" instead of "If none of the transactions is active". Thank you for bringing it to my attention.

Jayan said...

Hi, Anup
Very interesting and good to know about ITL. Eagerly waiting for another installment.

Thank you for your article.

NEERAJ VISHEN said...

Hi Arup

You are simply superb.........
what a neat and nice explanation ....

Thank you for your good post ....

Robert Bevan said...

After searching for this information, I will have to say most people agree with you on this topic.

gautham said...

Arup, Is there information in the ITL slot which shows which row is being locked?I see in your diagram that you have linked ITL1 slot with record 1 but in reality, is this for only for understanding purposes or does the row show the ITL it is linked to or does the ITL show the row it is linked to(i don't see this info though)?

Ashfaqh said...

Its an excellent post !!
Thanks Arup.

Anand Chavan said...

Thanks Arup for this article.It is really useful and informative.Nice explanation in a simple language.

Girish said...

Absolutely excellent article with informative examples

Anonymous said...

Dear Arup,

please describe the details of undo segment (Undo Segment Number,Slot# in the undo segment,Sequence# (or wrap) in the undo slot).

I can not find any docs about it.

Thanks
K.P

meldaresearch said...

Getting reliable and competent Professional Proposal Writers is no longer an easy task as most of the current providers are unreliable and provide poor Nursing Essay Writing Service that do not let students get high scores.

meldaresearch said...

Research Papers Writing Services should strive to ensure that all essays meets the pukka standards of the examining bodies. Consider hiring quality but also Qualified Research Paper Writers which are advantageous in terms of ensuring you get only top-quality grades on your Online Research Paper Writing Service.

Formed cru said...

Waooow!! Nice blog, this will be greatly helpful.

automatic screw locking machine

Unknown said...

I can't believe I can earn money weekly from trading , this is amazing , and all this is from the effort of a company called skylink technology whom I met online and help me out in trading and gave me good tips about trading physiology... indeed skylink technology is a bitcoin/binary forex experts and company and I won't stop thanking them and sharing my testimony until am fully satisfied...... Interested traders should  free free to contact mail: skylinktechnes@yahoo.com  or  whatsapp/telegram: +1(213)785-1553 

chanee said...

A good website is useful.
universocampo.com/
searchjobscareers.com/

majortotositepro said...

This is very interesting, You are a very skilled blogger. 메이저사이트

Canon Printer Error Code 5200 said...

This is a great article, with a lot of information in it, These types of articles make users interested in your site. Keep sharing more of the better posts! Thanks!

yadongbiz said...



Thanks For Sharing Such An Interesting Article, It Is Really Worthy To Read. I Have Subscribed To You And From Now On I'll Check Your Profile Daily For Interesting Stuffs. Feel free to visit my website; 야설

yahanvideonet said...

I must thank you for the efforts you’ve put in writing this blog. I am hoping to view the same high-grade blog posts from you later on as well. In fact, your creative writing abilities has inspired me to get my very own blog now ?? Thank you for the auspicious writeup. Feel free to visit my website;
일본야동


koreayadongcom said...

I am really amazed at the ideal approach to viewing your subject. Where do you collect this information? I want to write great writing like you do. Feel free to visit my website; Feel free to visit my website;

국산야동

chinayadongnet said...


Your article has proven useful to me. It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content. Feel free to visit my website;
일본야동

japanyadongcom said...


Efficiently written information. It will be profitable to anybody who utilizes it, counting me. Keep up the good work. For certain I will review out m ore posts day in and day out. Feel free to visit my website;
한국야동

Majortotosite Pro said...

Wonderful post! We will be linking to this great article on our site. I’m definitely delighted I found it and I’ll be bookmarking and checking back often! 토토

John Miller said...

Thanks for sharing great post!
Best SEO Company in India

Tang San said...

This article gives the light in which we can observe the reality. This is very nice one and gives indepth information. Thanks for this nice article.สล็อตออนไลน์

Tang San said...


Positive site, where did u come up with the information on this posting? I'm pleased I discovered it though, ill be checking back soon to find out what additional posts you include.สล็อต 999

Tang San said...

Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts.
สล็อตวอเลท

Tang San said...

Positive site, where did u come up with the information on this posting?I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work.
สล็อตแตกง่าย

Tang San said...

I just want to let you know that I just check out your site and I find it very interesting and informative..
บา คา ร่า วอ เลท

Tang San said...



I am very much pleased with the contents you have mentioned. I wanted to thank you for this great article.บา คา ร่า วอ เลท

Tang San said...



I really loved reading your blog. It was very well authored and easy to understand. Unlike other blogs I have read which are really not that good.Thanks alot!สล็อตเว็บใหญ่

Tang San said...


You have a good point here!I totally agree with what you have said!!Thanks for sharing your views...hope more people will read this article!!!
บา คา ร่า วอ เลท

Tang San said...


You’ve got some interesting points in this article. I would have never considered any of these if I didn’t come across this. Thanks!.สล็อต ฝาก-ถอน true wallet ไม่มี บัญชีธนาคาร

Tang San said...


This blog is really great. The information here will surely be of some help to me. Thanks!.บาคาร่าวอเลท

Tang San said...


hello!! Very interesting discussion glad that I came across such informative post. Keep up the good work friend. Glad to be part of your net community.
สล็อตแตกง่าย

Tang San said...



Thanks for sharing this quality information with us. I really enjoyed reading. Will surely going to share this URL with my friends.สล็อตxo

Tang San said...


This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!.เว็บสล็อต

Tang San said...



This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post.เว็บสล็อตเว็บตรง

Tang San said...

Nice to be visiting your blog again, it has been months for me. Well this article that i’ve been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share.
เว็บ ตรง

Tang San said...

This is a great article thanks for sharing this informative information. I will visit your blog regularly for some latest post. I will visit your blog regularly for Some latest post.
I'm glad to see the great detail here!.เกมสล็อต

Tang San said...

It was a very good post indeed. I thoroughly enjoyed reading it in my lunch time. Will surelycome and visit this blog more often. Thanks for sharing เว็บตรงสล็อต

eddielydon said...

Great information about wilderness for beginners giving the opportunity for new people. Hells Angels Jacket

horror halloween outfits said...

You have made this article in an exceptionally imaginative way. horror halloween outfits

Filmsjackets said...

Highly appreciate your Motorcycle Jackets work and research. Thank you

Translate