The PASCAL Object Recognition Database Collection
Caltech 101 and 256 labeled pictures of objects belonging to 101/256 object categories.
Image Net contains images of 10,000 object categories organized in a semantic hierarchy.
MIT databases for faces, pedestrians and cars
GRAZ 02 database of segmentation masks for bikes people and cars
UIUC Image Database for Car Detection
ETH-80 database for object categorization
Wordnet A searchable database of the relationships between different words. Useful for automatically determining what image annotations mean.
Flickr has a large collection of searchable images which can be automatically downloaded
Aerial images of the entire California coastline are available here.
The Computer Vision Homepage contains an extensive list of other databases