I have been trying to invade a system but can not seam to work out how.

Every turn a new ship shows up and when I try invade it says "you can not invade a defended planet" I can now work out how to attack the ship, unless I hit the "engagde system" button. Then the turn instantly ends and next turn there is a new ship there.

When you click on the system and see the list of ships including the enemy, how do you fight them? Also how to you invade a system?

If a planet is defended it has either ground or orbital defenses or ships (theirs or their allies). If you engage and nothing happens, it's likely because they aren't hostile to you. In this case you need to manually move a ship to attack first before you auto-resolve (if you want the AI to handle it). Note that light beam weapons aren't effective on planetary defenses.

Once a planet is no longer defended (it will show blockaded), you can then give an invade order.

but how do you do that?

The only way I have worked out how to have combat is to click "engage system". This instantly ends the turn, and then I have a combat with one fleet at that system. I kill the fleet, and then next turn there is a brand new one there. I assume built that turn using "buy out". It is some kind of loop. I can never seam to attack the planet as there is always the words "defended".

Is there any other way to get into combat with out ending the turn?

My fleet has a bunch of Size 2 ships loaded with Ion Cannons and 2 size 3 ships with troops and bombs and a few ion cannons.

Also I am playing classic, not "normal".

With instanced combat you should be seeing a prompt to invade right after attacking a planet (after you defeat the ship defending it/it's defenses) if your ships have soldiers.

If you can send a report (from the launcher) I can take a quick look of your saved game to see what's going on exactly.