Wednesday, February 09, 2005

Introduction to Linux Device Driver Programming

Author: Gaurav Dhiman
Email: gauravd_linux@yahoo.com, gaurav4lkg@gmail.com


Introduction to Linux Device Drivers:

Linux Device Driver is actually the peace of code which very well knows the device it is controlling. It knows the behavior and has knowledge of device internals. Device Drivers in Linux can be a part of core kernel it self or it can even be developed as a separate module, which can be attached/detached from running kernel anytime, providing a flexibility in kernel to support multiple devices in dynamic environment.

In this article we will talk about writing a device driver as a kernel module. Before talking about device drivers, we should have some basic knowledge of kernel module programming in Linux. In next few sections we will discuss the basics of kernel module programming before jumping to the driver intricacies.


Introduction to Kernel Module Programming:


Kernel module is a piece of code written, compiled and loaded separately from core kernel but linked to the core kernel at load time. Kernel modules have some specific structure to follow. There are two standard functions which need to be implemented in any Linux Kernel Module; we will talk about them bit later. As earlier also mentioned kernel module is a code which attaches to the core kernel at load time and is being executed in kernel or privileged mode. Its not a user program which runs in restricted mode and have limited access to memory and other system resources. Module being a part of the kernel can access any system resource. Due to this fact, kernel module programmers need to take care that their module does not create the security loopholes in kernel and must be well-disciplined. Kernel modules should only be doing what they are supposed to do. One loosely written kernel module loaded to the core kernel can make the whole system venerable.

Module structure:
Module need to have two standard functions which take care of initializing and cleaning up of resources used by module. The standard format of module is as follows:

#include
#include

/*standard function for initializing the resources used by module*/

int init_module(void){
}

/*other function, which mplements the functionality of module*/

func_1(){
……..
……..
}

func_2(){
………
………
}

func_3(){
………
……..
}

……..
…….
………

/*standard function for initializing the resources used by module*/

void cleanup_module(void){
}

In init_module() function, we request and acquire the resources required by our module. For e.g. resources can be an IRQ line (interrupt line on which our device is going to interrupt), I/O memory region, DMA channel etc. We also register few other things like major number used by our driver or the interrupt handlers for the interrupts generated by our device. Init_module() function is called at the time of loading a module to core kernel, using “insmod” or “modprobe” command.

In cleanup_module() function, we release the resources acquired in init_module() function. This function is called at the time uloading a module from core kernel, using “rmmod” command.


Presenting real time Kernel Module:
Let’s follow the tradition and write the first kernel module “hello_world.c”

#define __KERNEL__
#define MODULE

#include
#include

int init_module(void){
printk(<1> “My Module: Hello World !! \n”);
}

void cleanup_module(void ){
printk(<1> “My Module: Bye World ….. I am going !! \n”);
}


We need to define above mentioned two macros (__KERNEL and MODULE) for any module we write. printk() function is a brother of printf() function. The major difference between them is that printf() is a standard C library function which resides in user space and printk() is kernel function written from scratch.


Now coming the real thing – Device Drivers
Before going to the real code of device driver, let’s first understand the some basic fundamentals of what devices are in Linux. We will also discuss about how they are accessed by user processes.


Device Files and Major Numbers:
In Linux or and other Unix variant, devices re represented by files in file system. These files are known as device files or even nodes. Device files are special files through with the user process (program) can communicate (open, read, write, close etc) with the underlining device. Process uses the standard system calls, like open(), read(), write(), ioctl() etc to interact with device. With every device file a two special number are associated, which are known as a major number and minor number of a device. Major number is used by kernel to direct the user process request to right kernel driver and minor number is useless for kernel. Minor number is only used by the driver to identify the exact device which needs to be manipulated. The reason of existence of minor number is that, in practical scenario one driver can handle or control more than one device of same type or even of different types as well, so driver needs some mechanism through which it can identify which device it need to manipulate. Lets take an example, if there is a driver ”A”, which actually controls three physical devices “B”, “C” and “D”, then there need to be three device files in file system with same major number but different minor numbers.

Device files can be created with “mknod” command. It has the following syntax.

mknod {device name} {device type} {major number} {minor number}

For help on this command, refer to man pages of it.
It is a convention that device files reside in “/dev” directory of root file system, so it’s always better to create our own device files there rather than creating them somewhere else in file system.


Device Types:
Devices can mainly be categorized in three groups: character devices, block devices and the network devices.

Character Devices: These are the devices to which a user process can write or read a single character at a time. That means the interaction between device driver and actual physical device is in terms of single character. Example: keyboard, serial ports, parallel ports.

Block Devices: These are the devices to which a user process can write or read a single block of data at a time. Reading and writing on these devices are done in terms of block data. A block can be of 512 bytes, 1024 bytes or so. Example: Hard Disk, Floppy Disk.

Network Devices: These are asynchronous devices and are responsible for establishing a network connection to outside world. Best example of this type of device can be NIC card.

For not complicating things, this article will only talk about the character device drivers.


Opening a Device:
As mentioned earlier also that user process can communicate with the underlining device through device file, so for interacting with device, user process should first open the related device file, using open() system call. Once the device file is opened, user process receives the file descriptor, which it can refer in further file manipulation file system calls.

Now we will see how things work in kernel when a device file is opened by a user process. Before discussing it, let’s discuss about some related data structures.

task_struct: This data structure represents the process or task in kernel. Kernel uses this data structure to keep track of process in system and resources they are using. It is one of the main data structures in kernel and contains number of elements to track process specific information.

files_struct: This structure contains the information related to the open files per process. It keeps information to track the open files of a process. One of the elements of this structure is an array of pointers to “file” structure. This array points to different “file” structures and the index of this array is returned to process as a file descriptor when open system call is made. It also keep a count of total number of open files for a process.

file: This structure represents an open file for a process. Do not confuse it with physical file, which is represented by “inode” structure in kernel. This structure only remains in kernel memory till the file is open for a process. As soon as the process closes, exits or aborts, all the “file” structures (representing open files for that process) are destroyed, if those are not anymore pointed by any other process.

Some of the elements of “file” structure:

- f_mode: This element tells in what mode the file has been opened by the process. Process can open a file in either read (FMODE_READ) or write (FMODE_WRITE) mode. This element is normally used by device drivers to check in what mode the device file has been opened.

- f_pos: This element tells the offset (in bytes) from where to read and write in file.
- f_flags: This element also tells us in which mode the file has been opened (read or write), but it’s always recommendable to use “f_mode” element to check the mode of file. This element remembers one important thing, which might be helpful to driver writers and that is if the file has been opened in non-blocking mode or not. By default (if O_NONBLOCK flag is not mentioned at opening time) the file is opened in blocking mode. Driver checks this flag at the time of reading or writing a device. If the device file is opened in blocking mode (default mode) and at read time there is no data to be read from device or at write time driver specific buffer is full, driver puts the process to sleep on one of its local wait queues (we will soon see what wait queue are). But on other hand if the device file has been opened in non-blocking mode then the driver does not put the process to sleep, rather control returns back to user process with error.
- f_count: This element keeps track of how many processes are referring to this instance of file. As we know that all files of parent process are inherited by child process if file does not have close_on_exec element set. If the child process inherits the files from parent process, the “f_count” element of all inherited files is incremented, so that kernel can keep track of number of process this file structure. “file” structure (representing an open file) does not get destroyed on all close system calls, as it might be shared by other process also. During close system call kernel checks the “f_count” element of “file” structure and if it is zero then only “file” structure is released and its memory Is released.
- f_owner: This element tells which process is the owner of this open file. It contains the pid of the owner process. This element is used for sending the SIGIO signal to the right process in case of asynchronous notification. User process can change this element by using fcntl() system call.
- f_op: This is an important element from the perspective of device driver. This element actually points to the structure of pointers to file operations. For a device file (represented by “file” structure), this element points to the structure, which further contains pointers to driver specific functions. We will discuss in detail, the structure (file_operations) to which this element points.


- file_operations: This is an important data structure for device driver, as this is the structure through which driver registers its functions with kernel, so that kernel can call them on different events, like opening a device file, reading/writing a device file or sending ioctl commands to device. In case of device file, this structure contains pointer to different functions of driver, through which kernel invokes the driver. Now we will briefly discuss the elements of this data structure.


Some of the elements of “file_operations” structure

- llseek: This is a pointer to driver function, which actually moves or sets the “f_pos” element of device file (discussed earlier).
- read: This is a pointer to driver function, which actually physically reads data from device.
- write: This is a pointer to driver function, which actually physically writes data to device.
- poll: This function is called by either “poll” or “select” system calls.
- ioctl: This function is called by ioctl system call. This function in drier is used to pass on the special commands to device, format the device or setting the read/write head of device, which are different from normal read / write commands.
- mmap: This function of driver is used to map the device memory area to process virtual address space.
- open: This function is used to open a file. Incase of device file, this function of driver initialize the device or initializes other book keeping data structures.
- flush: This function flushes the driver buffer to physical file. This should be implemented in driver if driver wants to provide a facility to application to make sure that all the data is physically put on device.
- release: This function is called by close system call, but it is not called for every close system call. As described earlier also that one “file” data structure (which represents an open file in kernel) can be referred by more than one process, if processes are sharing the file (best example is FIFO files), in that case close system file does not release the “file” data structure and only decrement the “f_count” element of “file” data structure. If after decrementing “f_count” element turns to be zero, close system call, calls the associated “release” function, which is a driver function in case of device files. So “release” function of driver should clear and release all the memory acquired. “release” is just an opposite of “open” function.


Few Important Kernel Mechanisms used in Drivers

Wait Queues:
Wait queue is a mechanism in kernel through which the kernel code can put the process to sleep. This is used in different parts of kernel where the kernel decides to put the process to sleep. Kernel puts the process to sleep in case the required event has not yet occurred (for e.g. some process wants to read from device and there is no data to be read), in this case kernel puts the process to sleep and gets back the processor from it by calling the schedule() function, which is a scheduler in Linux Kernel. schedule() function schedules and dispatches the other process.

Before discussing the function related to sleeping a process, we should look what data structures are used for implementing a wait queue in kernel.

Wait queue is actually a linked list of “wait_queue_t” type of structures. The head of wait queue is represented by “wait_queue_head_t” structure, which contains the spin lock to synchronize the access to wait queue. “wait_queue_head_t” structure also contains the pointer to the first element in wait queue. Each element in the wait queue is represented by “wait_queue_t” structure, which contains the pointer to the “task_struct” type of structure. It also contains the pointer to next element in the wait queue. “task_struct” represents the alive process in kernel. So with this mechanism of wait queue driver or any kernel part can keep track of process waiting for a specific event to occur.


Putting process to sleep:
Process can be put to sleep by using any of the following kernel functions. You can call these functions from anywhere in the kernel (drivers, modules or the core kernel) in case you want to put your process to sleep. Whenever a kernel code is executed (when system call is made by the user process), kernel code executes in the context of process which has made a system call. But there is exception to this rule, whenever the interrupt occurs the kernel code (interrupt handler) does not execute in process context, it’s a anonymous context. This is the reason that we should be careful to not to call any function in interrupt handler which can put the execution thread to sleep. If we do so the kernel will hang, that means the system will hang.

Functions which can put a process to sleep:
- sleep_on(wait_queue_head_t * wait_queue)
- interruptible_sleep_on(wait_queue_head_t * wait_queue)
- sleep_on_timeout(wait_queue_head_t * wait_queue, long timeout)
- interruptible_sleep_on_timeout(wait_queue_head_t * wait_queue, long timeout)

In above functions, “wait_queue” is the wait_queue_head and “timeout” is the value mentioned in terms of jiffies. We will talk about jiffies very soon. Now we will see the difference between above mentioned functions.

- sleep_on: This function puts the process to sleep in TASK_UNINTERRPTIBLE mode, which means the process will not be waked up in case process receives any signal while it was in sleep. The process will only be waked up any other part of kernel code wakes it up (normally on the occurrence of some event) deliberately by calling any of the waking function (we will be discussing the waking up functions very soon). Process put to sleep with this function can sometimes cause some problem. For e.g. if a process is put to sleep with this function and the event on which it need to be waked up does not occur then your process will not come back to the execution stage. That process can not even be killed by sending a KILL signal, as process in sleep in TASK_UNINTERRUPTIBLE mode ignores all signals. Process put to sleep with this function can be waked if any of the following conditions occur:

o Process is deliberately waked up by some part of the kernel code on the occurrence of event for which it was waiting

- interruptible_sleep_on: This function in kernel is written to avoid the problem caused by “sleep_on” function. This function puts the process to sleep in TASK_INTERRUPTIBLE mode. When a process sleeps in this mode, it can be waked up if any of the following condition occurs:

o Process receives the signal either from any other process or kernel itself.
o Process is deliberately waked up by some part of the kernel code on the occurrence of event for which it was waiting.

- sleep_on_timeout: This function is similar to “sleep_on” function but is not that much dangerous as “sleep_on”. Process put to sleep with this function can be waked if any of the following conditions occurs:

o Time mentioned in the timeout parameter has expired
o Process is deliberately waked up by some part of the kernel code on the occurrence of event for which it was waiting.

- interruptible_sleep_on_timeout: I hope by now you can easily guess what this function does. Well the process put to sleep with this function is waked up when any of the following conditions occurs:

o Process receives the signal either from any other process or kernel itself.
o Time mentioned in the timeout parameter has expired
o Process is deliberately waked up by some part of the kernel code on the occurrence of event for which it was waiting.

Waking up a process:
Process put to sleep should also be waked up by some kernel code else the process will never return to the execution state. If your driver is putting the process to sleep, it’s the responsibility of that driver itself to wake up the sleeping processes when the required event occurs for which those processes are waiting. For e.g. if your driver put the reading process to sleep on its internal waiting queue, if there is nothing to read from driver buffer (driver buffer empty) then the process put to sleep should also be waked up whoever new data arrives in driver buffer (this will occur when device interrupts, so interrupt handler will be responsible for waking up the process sleeping on the driver’s waiting queue).

Functions which can be explicitly called to wake up the process:
- wake_up(wait_queue_head_t * wait_queue)
- wake_up_interruptible(wait_queue_head_t * wait_queue)
- wake_up_sync(wait_queue_head_t * wait_queue)
- wake_up_interruptible_sync(wait_queue_head_t * wait_queue)

__________________________________________________________________

That’s it for this time. In next article I will cover ‘ioctl’ interface of devices and different timing mechanisms available in Linux Kernel.


46 comments:

Accounting software said...

Hi Gaurav Dhiman, I was just blog surfing and found you! Wow, I really like this one.
It’s such a pleasure to read your post …. Interesting! I was over at another site

looking at pos system

and they didn't go into as much detail as you, but nonetheless interesting.

ecommerce software said...

Hey this blog is not about ebusiness software

I have been doing hours of research on "Accounting-Software" and it brought me to your blog on Introduction to Linux Device Driver Programming. Anyways, Gaurav Dhiman I was reading your blog and I think it is really cool. It’s really a pleasure reading your posts! Keep up the great work.

Keep blogging away :-)

ebusiness software said...

Hey, you have a great blog here! I'm definitely going to bookmark you!
I have a Accounting software site/blog. It pretty much covers Accounting software related stuff.
Come and check it out if you get time :-)

Elwe Thingol said...

Thanks for providing such information. Though I do not programme device driver, it, indeed, illustrates some useful points for a newbie who are interested in kernel.

0xDEADBEEF said...

Hey Gaurav, it's nice to finally find a blog that's focused on Linux kernel development. It's very educational and enlightening indeed. This'd be very helpful gor a kernel newbie like me.

Go ahead, I am eager to see your next article.

Best Wishes

Anonymous said...

Hi Gaurav,

when i was searching for some of the system calls related to linux kernel,i found ur article on Linux module programming... and it is very interesting to read even a novice can understand how the modular programming is taking place in case of linux...keep up the good work and looking forward to see more articles from you....

Best wishes,
Vamsi.

Anonymous said...

Hi Gaurav, your article is marvellous !! I know there are many good books available to understand device driver programming.. but it needs a lot of efforts to scroll through the pages of those books.. and you article comes in here just to save people like me.. very well written.. i could understand now well.. Please keep your good work always up.. we need many more articles like this.. With Best Wishes, your friend

navneetgarg120 said...

I have read the same article in maurice j bache. Is it from the same book?

Anonymous said...

Hey, while searching for widgets for my blog, I stumbled upon www.widgetmate.com and wow! I found what I wanted. A cool news widget. My blog is now showing latest news with title, description and images. Took just few minutes to add. Awesome!

Manoj said...

Excellent job !!
Keep going !

totan said...

HALO SIR I AM A FRESH SOFTWARE ENGINEER AND HAVE LEAST IDEA REGARDING LINUX .... CAN YOU SUGGEST ME HOW SHOULD I PROCEED...MY MAIL ID IS anirban.ghatak@hotmail.com

Anonymous said...

I found this site using [url=http://google.com]google.com[/url] And i want to thank you for your work. You have done really very good site. Great work, great site! Thank you!

Sorry for offtopic

Anonymous said...

Who knows where to download XRumer 5.0 Palladium?
Help, please. All recommend this program to effectively advertise on the Internet, this is the best program!

Anonymous said...

How can i wipe out windows xp from my laptop and reinstall windows Me -the laptops native software?
I have recently bought a familiar laptop that is old. The person I had bought it from had installed windows xp on it, regular for all that it instance came with windows Me. I after to remove the windows xp because it runs slows on the laptop because it takes up more tribute than the windows Me would. Also I want to massacre windows xp because it is an forbidden copy. So when I tried to stir one's stumps updates on it, windows would not initiate updates because the windows xp is not genuine. [URL=http://cocahom.hostific.com]aftermarket car parts 1990 ford ltd[/URL]
----------------------------------------------------------------------

Answers :

It's best to take one's leave of [URL=http://ltufakb.hostific.com/jaguar-cichlid.html]jaguar cichlid[/URL] Windows XP and good upgrade your laptop. It's much better. [URL=http://xoqzfgh.hostific.com/g-pat-macha.html]g pat macha[/URL] Besides, Windows XP is style [URL=http://xoqzfgh.hostific.com/ga-unemployment-benifits-estimator.html]ga unemployment benifits estimator[/URL] heartier then Windows Me. Windows Me is obsolete and many programs that can hustle with XP, can't [URL=http://kouuvvs.hostific.com/mark-hipfner.html]mark hipfner[/URL] run with Me.
------------------------------
all you possess to do is brochure the windows me disk into the cd drive. then reboot your laptop, when the coal-black [URL=http://noxynjc.hostific.com/mbt-high-sports-shoe.html]mbt high sports shoe[/URL] sieve with all the info comes up and when it asks u to boot from cd [URL=http://zxrengh.hostific.com/robert-pettigrew-vancouver.html]robert pettigrew vancouver[/URL] belt any key when it tells you to then install from there !!! I RECOMEND SINCE ITS AN ILLEAGLE TWIN TO WIPE [URL=http://aoryvuo.hostific.com/trailer-stake-pockets.html]trailer stake pockets[/URL] OUT OF THE CLOSET THE [URL=http://uyyfodi.hostific.com/newberg-or-soil.html]newberg or soil[/URL] THOROUGH MAGISTERIAL SEND WHEN IT ASKS YOU WHICH IMPENETRABLE [URL=http://zanqmez.hostific.com/mitigation-skills-for-indonesia-tsunami-2004.html]mitigation skills for indonesia tsunami 2004[/URL] PUSH TO INSTALL IT ON. THEN ADD ALL THE ABOVE SPELL ON THE EMPTY [URL=http://onssiyd.hostific.com/imp-baby-clothes.html]imp baby clothes[/URL] URGENTLY CONSTRAIN ONTO A DIFFERENT DOCUMENTATION FINGERS ON, IT WILL-POWER LOOK LIKE C:/ Open or something like that

Anonymous said...

Hello
Companies House Webcheck
Webcheck Companies House
Company House Webcheck
Companies House Webcheck Service

[url=http://perosnalbinking.v3host.be/companies-house-webcheck.html]companies house webcheck service[/url]

Anonymous said...

picture piano london singles night [url=http://loveepicentre.com/]online free dating[/url] dating coffee jars http://loveepicentre.com/ free adult personals web cam sevice

Anonymous said...

excellent laptop battery [url=http://www.hqlaptopbatteries.com/-dv7-1170-laptopbatterymodel1482.html]best laptops[/url] compare laptops http://www.hqlaptopbatteries.com/-3660-laptopbatterymodel1455.html Apple laptop battery
original laptop batteries [url=http://www.hqlaptopbatteries.com/battery-7110-batterytype1.html]Apple laptop battery[/url] notebook batteries http://www.hqlaptopbatteries.com/-aoa110-laptopbatterymodel1588.html Apple Laptop
toshiba laptop batteries [url=http://www.hqlaptopbatteries.com/-z53-laptopbatterymodel872.html]Acer Laptop[/url] Asus Laptop http://www.hqlaptopbatteries.com/-g20-laptopbatterymodel1896.html IBM laptop battery

Anonymous said...

viagra affiliate [url=http://usadrugstoretoday.com/products/citalopram.htm]citalopram[/url] medical fundraisers http://usadrugstoretoday.com/products/hair-loss-cream.htm cc stands for in medicine http://usadrugstoretoday.com/products/zofran.htm
medicine online [url=http://usadrugstoretoday.com/categories/anti-diabetici.htm]anti diabetici[/url] health provider responsibility kansas [url=http://usadrugstoretoday.com/products/lozol.htm]informatoin on smoking[/url]

Anonymous said...

glass mineral blends of plastic to eliminate warpage [url=http://usadrugstoretoday.com/products/coreg.htm]coreg[/url] hugr breast http://usadrugstoretoday.com/categories/la-salute-delle-donne.htm dog anxiety medicine http://usadrugstoretoday.com/products/starlix.htm
va medical center ohio [url=http://usadrugstoretoday.com/index.php?lng=en&cv=us]Buy generic and brand medications[/url] dog withyeast infection [url=http://usadrugstoretoday.com/products/dramamine.htm]smoking fresh ham[/url]

Anonymous said...

http://www.apebfr.org/forum/viewtopic.php?p=126325#126325 http://www.zeuspaul.com/forum/viewtopic.php?f=2&t=186426 http://cn.bcwelt.de/viewtopic.php?f=8&t=10511 http://www.buildmasters.com.au/forum/viewtopic.php?f=9&t=330039
http://www.ksbamerica.com/board/zboard.php?id=calendar_la&page=1&page_num=20&select_arrange=headnum&desc=&sn=off&ss=on&sc=on&keyword=&no=13740&category=1 http://www.liog.org/Forum/index.php?topic=75095.new#new http://fadingendlessly.com/forum/viewtopic.php?f=4&t=13468 http://www.bravecave.com/forum/viewtopic.php?f=7&t=4277
http://www.mixit.co.nz/forum/viewtopic.php?f=2&t=121936 http://www.hawaso.com/JeilStudy/boardmodify.asp?BoardName=LectureQna&Num=2074&ScrollAction=Page6 http://spamswamp.com/index/viewtopic.php?f=4&t=54692

Anonymous said...

tea party ideas [url=http://usadrugstoretoday.com/products/hoodia-weight-loss-gum.htm]hoodia weight loss gum[/url] medical blogging liability http://usadrugstoretoday.com/categories/antivirali.htm gay man penis contest http://usadrugstoretoday.com/products/imuran.htm
where do i get a drug screening [url=http://usadrugstoretoday.com/products/generic-keflex.htm]generic keflex[/url] basic diabetic diet [url=http://usadrugstoretoday.com/catalogue/s.htm]tea for colds[/url]

Anonymous said...

http://healthboard.in/bodybuilding/male-bodybuilding-over-40
[url=http://healthboard.in/caffeine]forced drug rehabilitation wikipedia[/url] erectile dysfunction help [url=http://healthboard.in/coreg/does-tricare-pay-for-coreg]does tricare pay for coreg[/url]
decorative medicine cabinets in new york city http://healthboard.in/buspirone/inderal-difference-buspirone
[url=http://healthboard.in/cyproheptadine/cyproheptadine-for-felines]blue shield of california mail order drug program[/url] random drug testing effective [url=http://healthboard.in/cymbalta/highler-dosage-of-cymbalta-for-trd]highler dosage of cymbalta for trd[/url]
drug values clarification http://healthboard.in/calcium/calcium-pyrovate
[url=http://healthboard.in/cytotec/prescription-drug-cytotec]viagra contra cialis[/url] southlight drug raleigh [url=http://healthboard.in/carbohydrate/one-gram-carbohydrate-yeild-kcal]one gram carbohydrate yeild kcal[/url] pharmacy no presciptions [url=http://healthboard.in/bupropion/bupropion-dopamine]bupropion dopamine[/url]

Anonymous said...

al gore automobile http://automotoportal.in/motorcycle-show/motorcycle-helmets-custom-paint daytona beach automobile attractions
[url=http://automotoportal.in/lotus/lotus-land-cattery-russia]volkswagen jetta gls tdi used cars[/url] volkswagen chassis parts [url=http://automotoportal.in/panoz/panoz-gts]panoz gts[/url]
volvo automobile website http://automotoportal.in/skoda/concesionario-skoda-en-barcelona
[url=http://automotoportal.in/motor-co/fine-motor-skillls-and-color-preference]pictures of people in automobile accidents[/url] volkswagen golf turbo diesel pump performance tuning [url=http://automotoportal.in/morgan/morgan-auctions]morgan auctions[/url]
automobile repair service http://automotoportal.in/lexus/lexus-lx570-2008
[url=http://automotoportal.in/motors/emerson-5-hp-electric-motors]what is emission workshop on a volkswagen passat[/url] auto insurance providers in massachusetts [url=http://automotoportal.in/skoda/skoda-superb]skoda superb[/url]

Anonymous said...

travel vacation rimini http://livetravel.in/plane-tickets/immediate-international-plane-tickets travel when pregnant
[url=http://livetravel.in/lufthansa/boeing-spectralab]things to bring during travel internationally[/url] jewelry travel stores [url=http://livetravel.in/tourism/singapore-tourism-world-bank]singapore tourism world bank[/url]
travel agents in kodaikanal http://livetravel.in/tours/coupons-for-scenic-air-grand-canyon-tours
[url=http://livetravel.in/hotel/hotel-of-snake-river]airline employee germany discount travel[/url] possible christian travel business names [url=http://livetravel.in/disneyland/kaiser-alumonum-in-disneyland]kaiser alumonum in disneyland[/url]
hallmark travel list of residence http://livetravel.in/tours/candy-factory-tours-mn
[url=http://livetravel.in/inn/ramada-inn-denton]greece travel map htm[/url] travel warnings in jordan [url=http://livetravel.in/airlines/spirit-airlines-shop]spirit airlines shop[/url] las vegas miss america travel package [url=http://livetravel.in/tour/new-york-ellis-island-tour]new york ellis island tour[/url]
conn travel guide [url=http://livetravel.in/map/map-woodcliff-lakes-nj]map woodcliff lakes nj[/url]
poor people deals for air travel http://livetravel.in/cruise/norweigian-cruise-jobs
[url=http://livetravel.in/maps/fort-bragg-east-maps]all inclusice travel cancun riveria maya[/url] daytona 500 travel package [url=http://livetravel.in/hotel/alta-peruvian-hotel]alta peruvian hotel[/url]
[url=http://livetravel.in/adventure/adventure-science-museum-brentwood-tn]adventure science museum brentwood tn[/url] travel sheet [url=http://livetravel.in/adventure/adventure-holiday-chile]adventure holiday chile[/url] world travel map where been [url=http://livetravel.in/travel/nurse-travel-batesville-indiana]nurse travel batesville indiana[/url]
travel destination for swingers [url=http://livetravel.in/inn/holiday-inn-express-st-bernard]holiday inn express st bernard[/url]

Anonymous said...

travel agency templates http://atravel.in/motel_denton-tx-days-inn-motel belgium travel guide
[url=http://atravel.in/vacation-packages_cheap-vacation-packages-to-los-cabos-mexico]pet travel maryland[/url] amx travel [url=http://atravel.in/airport_car-hire-from-pisa-airport]car hire from pisa airport[/url]
west yellowstone travel http://atravel.in/vacation-packages_canada-discount-vacation-packages
[url=http://atravel.in/motel_libby-montana-motel]travel deals to florida[/url] travel trailer buyers guide [url=http://atravel.in/flight_microsoft-flight-simulator-accessories]microsoft flight simulator accessories[/url]
travel safety in venezuela with children http://atravel.in/maps_table-rock-maps csonka pocket travel ashtray [url=http://atravel.in/tourist_melbourne-tourist-info]melbourne tourist info[/url]

Anonymous said...

automobile payment calculator http://autoexpress.in/rally/daytona/spring/bike/rally volkswagen table lamp
[url=http://autoexpress.in/rover/land/rover/exchange]mercedes c240 wheels or rims[/url] best xbox 360 racing games [url=http://autoexpress.in/seat/stadium/seat/cushion/with/back/rest]stadium seat cushion with back rest[/url]
volkswagen city http://autoexpress.in/maserati/maserati/quattroporte/forum/jeep/grand/cherokee
[url=http://autoexpress.in/panoz]automobile tires cheap[/url] kaase auto polish [url=http://autoexpress.in/buell/free/buell/xb12s/manual]free buell xb12s manual[/url]
drag racing in south africa http://autoexpress.in/scooter/kymco/500cc/scooter
[url=http://autoexpress.in/saturn/saturn/sky/redline/cold/air/intake]auto wheel cribs[/url] grand theft auto san andreas pc version like slow motion [url=http://autoexpress.in/radiator/griffen/radiator]griffen radiator[/url]

Anonymous said...

weddingclothes http://topcitystyle.com/?action=products&product_id=2297 famous fashion designers [url=http://topcitystyle.com/?action=products&product_id=295]fashion in the sixties[/url] how to lace shoes differently
http://topcitystyle.com/shoes-new-category12.html arnold palmer augusta shoes [url=http://topcitystyle.com/philipp-plein-women-apos-s-long-sleeve-tops-brand31.html]author stephanie laurens[/url]

Anonymous said...

grees fashion http://topcitystyle.com/abercrombie-and-fitch-polo-shirts-brand63.html martial arts shoes [url=http://topcitystyle.com/27-women-size17.html]nyc shopping designer knock offs[/url] traditional middle eastern clothes
http://topcitystyle.com/trussardi-jeans-brand11.html shimano road bike shoes [url=http://topcitystyle.com/?action=products&product_id=1921]pippen shoes[/url]

Anonymous said...

henschenklein worten http://topcitystyle.com/off-white-sports-shoes-color3.html tutorial of database designer [url=http://topcitystyle.com/multicolored-casual-shirts-color13.html]best paying fashion job italy[/url] extra wide shoes
http://topcitystyle.com/?action=products&product_id=1306 peart izumi spinning shoes [url=http://topcitystyle.com/44-winter-jackets-size26.html]ralph lauren advertisments who are models[/url]

Anonymous said...

insole shoes http://topcitystyle.com/hugo-boss-casual-brand34.html nike airmax shoes youth [url=http://topcitystyle.com/men-page33.html]womens harley davidson shoes[/url] taking clothes off
http://topcitystyle.com/dolce-amp-gabbana-bomber-leather-jacket-for--item2295.html make shoes [url=http://topcitystyle.com/yellow-underwear-color44.html]editions clothes[/url]

Anonymous said...

cmopetition mercedes http://carautonews.in/automobile/automobile-parts-fusible-links-for-dodge-trucks new volkswagen golf gt sport
[url=http://carautonews.in/citroen/david-fox-citroen]auto tbar exe[/url] mercedes india airbag disable [url=http://carautonews.in/bike/weslo-e-23-recumbent-bike]weslo e 23 recumbent bike[/url]
berreta auto shotgun http://carautonews.in/automobiles/fuel-savers-automobiles
[url=http://carautonews.in/bike-com]automobile training facilities design[/url] used volkswagen jetta diesel [url=http://carautonews.in/ferrari/robert-ferrari-turbine-entertainment-software]robert ferrari turbine entertainment software[/url]
automobile emanuals hyundai accent http://carautonews.in/audi/official-audi-car-dealer
[url=http://carautonews.in/bentley/bentley-continential-gt]volkswagen logo design[/url] napa auto repair [url=http://carautonews.in/auto-co/auto-paint-interior]auto paint interior[/url]

Anonymous said...

hentai part b scene hentai virgin auction http://planetofporn.in/erotic-gay/gat-erotic-stories
[url=http://planetofporn.in/erotic-video/mexican-woman-jpg-erotic]free black dildo movies[/url] preteen girls who dress sexy [url=http://planetofporn.in/free-amateur/amateur-teen-porn-movies-free]amateur teen porn movies free[/url]
adult clips 8 http://planetofporn.in/ass-video/asian-ass-porn-video
[url=http://planetofporn.in/adult-online/adult-film-savannah]sexy tighs butt[/url] teen with dildo [url=http://planetofporn.in/adult-movie/augusta-maine-adult-education]augusta maine adult education[/url]
virgin shemale sex pics http://planetofporn.in/fetish/gay-diaper-fetish
[url=http://planetofporn.in/adult-online/play-free-adult]anal pissing[/url] sexy junior models [url=http://planetofporn.in/hcg-oral/giving-cats-oral-drops]giving cats oral drops[/url]
teen denim porn http://planetofporn.in/archive-adult/adult-sluty-cheerleaders
[url=http://planetofporn.in/condom/jeweled-dildo]free classy porn[/url] submit amature adult video [url=http://planetofporn.in/adult-porn/adult-christmas-fun]adult christmas fun[/url]

Anonymous said...

sexy mature fat sluts http://theporncollection.in/porn-girl/free-porn-passwords-torrent
[url=http://theporncollection.in/porn-dvd/kelle-porn-marie]gisel amateur[/url] adult vide [url=http://theporncollection.in/gay-xxx/barely-legal-gay-sex-pics-galleries]barely legal gay sex pics galleries[/url]
myspace christmas graphics sexy http://theporncollection.in/lesbian-sex/gay-and-lesbian-authors
[url=http://theporncollection.in/orgasm/teen-tickling-orgasm]cheap adult novelties online[/url] hentai babe gets tortured [url=http://theporncollection.in/sex-mature/mature-women-sex-videos-free]mature women sex videos free[/url]
thin black teens anal http://theporncollection.in/sex-mature/uk-amateur-mature-xxxx-dp-bukakke
[url=http://theporncollection.in/free-hentai/flowers-of-hentai]virgin mobile v7 flasher cdm 8910[/url] hailey young anal vid [url=http://theporncollection.in/porn-dvd/free-schoolgirl-porn-galleries]free schoolgirl porn galleries[/url]
underage porn searches http://theporncollection.in/lesbian-porn/free-online-lesbian-sex
[url=http://theporncollection.in/lesbian-xxx/lesbian-doggie-style-sex]extra virgin olive oil[/url] mom son hentai [url=http://theporncollection.in/masturbating/nude-teens-masturbating-in-shower]nude teens masturbating in shower[/url]

Anonymous said...

gap insurance travel uk year http://xwa.in/tours/holt-tours los angeles travel package
[url=http://xwa.in/car-rental/car-rental-memphis-tennessee]bretagne travel hotels[/url] florida holiday travel solicitors [url=http://xwa.in/tourism/bvi-tourism-site]bvi tourism site[/url]
air travel health http://xwa.in/tourist/bedford-tourist-information-guide
[url=http://xwa.in/tours/rowan-yarn-factory-tours]travel mexico safety[/url] travel agents in greece [url=http://xwa.in/tour/bicycle-vacation-tour]bicycle vacation tour[/url]
lighted travel alarm clock http://xwa.in/tour/eco-tour-outdoor-adventure-travel-cheap-travel-insurance-uk ixtapa mexico travel deals [url=http://xwa.in/cruise/cruise-singles-cheap]cruise singles cheap[/url]

Anonymous said...

wild child new movie [url=http://worldmovs.co.cc/full_version-the-blues-brothers/10844database/]The Blues Brothers[/url] watch scary movie 4 free online [url=http://worldmovs.co.cc/full_version-piratas-del-caribe/826database/]Piratas Del Caribe[/url]
what movie did leonardo dicaprio first appear in [url=http://worldmovs.co.cc/full_version-the-incredible-adventures-of-wallace-and-gromit/29393database/]The Incredible Adventures Of Wallace And Gromit[/url] cars the movie website [url=http://full-length-movies.com/dvd-quality-movie-mystic-river/1317database/]Mystic River[/url]
bratz movie 17402 [url=http://full-length-movies.com/dvd-quality-movie-maneater/27664database/]Maneater[/url] surf movie wikipedia scene jets dick [url=http://full-length-movies.com/dvd-quality-movie-the-cradle/24124database/]The Cradle[/url]
most popular viv thomas movie [url=http://full-length-movies.com/dvd-quality-movie-one/7156database/]One[/url] movie with kelly preston and kevin costner [url=http://full-length-movies.com/dvd-quality-movie-highway-to-hell/24650database/]Highway To Hell[/url]

Anonymous said...

red tea kettle [url=http://usadrugstoretoday.com/products/pro-erex.htm]pro erex[/url] costs of hypertension or high blood pressure http://usadrugstoretoday.com/products/mevacor.htm
pill c03 [url=http://usadrugstoretoday.com/catalogue/y.htm]medications without a prescription[/url] pregnant breast are not soar [url=http://usadrugstoretoday.com/products/mobic.htm ]efficacy of drugs [/url] john edwards health care plan
lacatating breast milk movie free download [url=http://usadrugstoretoday.com/categories/sleeping-aid.htm]sleeping aid[/url] red blood in bowel movements http://usadrugstoretoday.com/products/generic-motrin.htm
photos of diet pills from the 1980s [url=http://usadrugstoretoday.com/products/viagra-professional.htm]viagra professional[/url] charles lyles diabetes [url=http://usadrugstoretoday.com/products/dipyridamole.htm ]high red blood cells bone marrow chemo [/url] vitamin d in osteoporosis

Anonymous said...

clothes internet shopping http://www.thefashionhouse.us/-leather-jacket-gucci-category85.html outdoor exchange clothes [url=http://www.thefashionhouse.us/black-casual-color2.html]giada delaurentis nude pictures[/url] tomasello shoes streator il
http://www.thefashionhouse.us/dolce-amp-gabbana-polo-shirt-for-men-sky--item2557.html gucci deigns [url=http://www.thefashionhouse.us/-maxi-skirt-women-category100.html]lauren motyl[/url]

Anonymous said...

travel clothes steamer http://www.thefashionhouse.us/women-roberto-cavalli-type3.html cheap baby clothes [url=http://www.thefashionhouse.us/cream-navy-blue-color162.html]compare yves saint laurent mascara volume no eff faux cils[/url] latest ladies fashion
http://www.thefashionhouse.us/burberry-sweaters-brand4.html how to wear red clothes [url=http://www.thefashionhouse.us/roberto-cavalli-dress-for-women-black-white-item2372.html]compare yves saint laurent mascara volume no eff faux cils[/url]

Anonymous said...

import street racing car http://xwm.in/buick/buick-skylark-patch bouncing email to a printer to print auto
[url=http://xwm.in/daewoo/daewoo-dbp-2000]mercedes classe c 2007[/url] pimp myspace mercedes [url=http://xwm.in/saleen/saleen-wings]saleen wings[/url]
volkswagen up http://xwm.in/chevrolet/chilton-chevrolet-avalanche-manual
[url=http://xwm.in/royce/the-best-of-rose-royce]colored auto painters tape[/url] volkswagen commercial van dealer perth [url=http://xwm.in/buick/land-rover-buick-v8]land rover buick v8[/url]
automobile in 1885 http://xwm.in/auto/no-park-position-in-bmw-auto-transmission
[url=http://xwm.in/yamaha/used-yamaha-for-sale]noble automobile[/url] newberry county career center auto [url=http://xwm.in/racing/sailing-anarchy-2007-kwrw-premier-racing-day-4-tent]sailing anarchy 2007 kwrw premier racing day 4 tent[/url]

Anonymous said...

blackjack cases http://lwv.in/gambling-online/best-gambling-list uk lottery results history
[url=http://lwv.in/online-casinos/ontario-casinos]potawatomi concerns gaming casinos[/url] the location for gambling state university [url=http://lwv.in/blackjack/casino-bonuses-blackjack-cards]casino bonuses blackjack cards[/url]
maine lottery raffle http://lwv.in/slot/counterstrike-10-slot-us-sever
[url=http://lwv.in/online-casinos/gold-coast-australia-casinos]betting on us collge football bowl games[/url] baccarat resrot anguilla [url=http://lwv.in/baccarat/baccarat-black]baccarat black[/url]
easy lottery http://lwv.in/casino-online/hairshows-at-the-soaring-eagle-casino gambling bonus how to play craps [url=http://lwv.in/roulette/pico-roulette-password]pico roulette password[/url]

Anonymous said...

titanic movie dialogue [url=http://moviestrawberry.com/films/film_superhero_movie/]superhero movie[/url] captain america movie offical website http://moviestrawberry.com/films/film_the_shining_70/ noorie hindi movie songs
movie featuring song born to be wild [url=http://moviestrawberry.com/films/film_industrial_britain/]industrial britain[/url] movie gallrey http://moviestrawberry.com/films/film_they_re_off/ final fantasy 7 movie
chinise movie trenslated english [url=http://moviestrawberry.com/films/film_alice_upside_down/]alice upside down[/url] not another teen movie cast
san luis obispo movie [url=http://moviestrawberry.com/films/film_the_fourth_protocol/]the fourth protocol[/url] movie theaters co http://moviestrawberry.com/films/film_whats_the_worst_that_could_happen/ cime city 2007 french movie
clear and present danger movie [url=http://moviestrawberry.com/films/film_robots/]robots[/url] trick or treat the movie http://moviestrawberry.com/films/film_the_santa_clause_2/ movie with mackenzie astin and ione skye

Anonymous said...

http://bluecrossblueshield.groupsite.com bluecross blueshield of florida

Anonymous said...

The emergence of little huge absolutely corporate, almost national, and international marketing chains and companies[url=http://www.freshgreens.info].[/url]

Anonymous said...

Helo ! Forex - Outwork чашкой чая успешно получать деньги, пройти регистрацию forex [url=http://foxfox.ifxworld.com/]forex[/url]

Anonymous said...

[url=http://sexrolikov.net.ua/tags/%C0%ED%E0%F2%EE%EB%FC%E5%E2%ED%E0/]Анатольевна[/url] Смотри и дрочи : [url=http://sexrolikov.net.ua/tags/%CF%E5%F2%E5%E9/]Петей[/url] , все это смотреть

Michale Daniel said...

Well It Was Very Good Information For Linux Learners.i Was Really Happy See This Information We Are Also Provide Linux Online Training All Courses.