Why are land taxes special?

Recently Bryan Caplan and I were chatting about Georgism.  Henry George is well-known for his insistence on taxing land.  But why land?  What makes land special rather than labor or capital?

As I understand the doctrine, there is no special case for taxing improvements on land.  Instead government should tax the "barebones" or "in situ" value on land.  Say that land currently sells for $100,000 an acre, but would sell for $50,000 unimproved.  We should levy the tax only on the $50,000.  Supposedly we are then taxing an inelastic factor and creating only minimal distortions.  Did not Adam Smith offer a similar recommendation?  What better way to fund government?

Fair enough, but then why not tax the in situ values of labor as well?  The "barebones" value of labor is of course leisure.  That is what labor is worth when no extra effort is added to the picture.  Therefore an optimal tax system should try to tax leisure.  This may prove difficult, but why should it be harder than taxing the barebones value of land?  Note that sometimes we are content to tax complements to leisure, such as large camper vans. 

What if we taxed complements to the in situ value of land?  These would be the factors -- like labor and capital -- that add value to barebones land.  So I take the Georgist view to imply two claims.  First, it is easier or better to tax barebones land than barebones labor.  Second, taxing a factor directly is much better than taxing complements of that factor.  Since I am not convinced either of these are true, I hold no particular attachment to the idea of a single tax on land.

I also share Benjamin Tucker's concern as to how the in situ value of land should be defined.  Some of the problems are conceptual rather than empirical.  Land values are interdependent.  When assessing the in situ value of my land, what assumptions should be made about the values of surrounding lands or the actions of other people?  When the steamship is invented, should all taxes on American land have risen? 

I might add that Bryan has different objections, which he may someday reveal to us all.