The Capital is the province where the country's seat of the government lies.
In Victoria 2 the capture of capital province increases War score significantly, and if it 's under control of rebels (other than nationalist) for 30 days, they will enforce their demands.
The Capital can not be taken with 'acquire' state or 'demand concession' casus belli. To take its capital the whole country has to be annexed with the 'annex' casus belli.
It is not possible to change a nations capital except in a few special cases. 
Russia and 
Italy have the decisions Return to the Third Rome and The Law of Guarantees respectively and The 
Ottoman Empire has the event Flight from Istanbul, which changes the capital.
A List of nations' capitals can be found at Country basic info. Note that if a nation is formed by decision, it will keep the capital of the forming nation regardless of the capital in files.