A plotline is very important to your custom battle or mission. Half of making a great battle is having a good plotline (the other half being good mission design). You already know this: after all, the same applies to movies and books. The first half is having a good story, the other half is actually making/writing it. If the plotline and creation work together, you end up with a great battle. If either is lacking, your battle is automatically condemned to the lower regions of the Compendium's quality chart.
Plotlines stretch as far as you can stretch your imagination. However, there are a few things to keep in mind at all times:
The first pilots who fly your missions will usually have flown a lot. They are looking for something new to catch their attention. If it's yet another battle where the only objective is to destroy a traitor with a few dozen T/Ds, it gets real boring real fast. The same applies to hunting down known Star Wars or game characters. We've all killed Zaarin and Han Solo a dozen times over already. While drawing inspiration from people never hurt anyone, there's a line between drawing inspiration and copying, a line which you should not be crossing at any time.
You can come up with the most fantastic plotline ever created, but if the platform for which you are creating cannot handle it, your mission is doomed. There are millions of things you can do, but millions is not the same as limitless. Of course things are a lot easier in XWA than X-wing, with allowing a lot more craft in the battle at any one time, and a lot more inflight messages, and other things, but even XWA has its limits (says the one who saw a great idea smashed because even XWA has a craft limit...) Be sure to figure out the limits on craft, flight groups and in flight messages before you start working on your missions!
Also important: make sure the pilot knows what he is supposed to be doing. Either make it so the briefing explains, or have the inflight messages explain, and use the mission goals wisely. Especially in XvT and XWA, where you can have custom goal tags, make sure they actually make sense. Having a mission goal that says "Have fun" sounds great, but might be very annoying because you don't have a clue what to do.
You will also have to make sure the mission works. Don't make it a mission goal to have a shuttle appear when the enemy beats an ISD down to 50% hull strength and then create a situation where a good pilot is able to keep the ISD safe from harm! Never should a pilot have to under-perform in order to win a mission.
as my English teacher always used to say. In other words: make sure you have a clear picture in your head when it comes to a storyline. It might even help to write it down on paper first. What happens in which mission, how do the missions tie together. The better your preparation is in this regard, the better your final product will be.
It's been said before, but the mission should be fun. There should be diversity in your missions. Making a 20 mission battle may seem like fun, but if every single one of those missions is "destroy a flight of Y-wings, hyper to next region, destroy a flight of A-wings, hyper to next region, destroy a capital ship, hyper home", it gets very old after 3 times. Look at the original missions: the missions in a battle tie together, but there is variation in them. And it should just simply be fun to fly. That also means it shouldn't be ridiculously hard on easy, or terribly easy on hard. Balance those difficulties! This also means you have to know the AI and set them right. And very importantly: try to stay away from invincible craft. If a craft may not be destroyed, make sure it hypers out before that happens.
If you've flown a lot of custom battles, you'll undoubtedly have run into missions where you are supposed to destroy everything. While on occasion there is no harm in this, make sure to avoid using it in every mission. It's boring, it's annoying, and will seriously harm the player's idea of your skills. It also adds to the realism factor. If you are to protect a capital ship, there is no reason to make it a primary goal to destroy all Rebel fighters. It makes for a nice bonus goal for those high score getters, but it sucks as a primary goal. Especially if the capital ship hypers out after 5 minutes and you then need another 10 minutes getting rid of all those fighters. Especially in TIE Fighter, such goals are dangerous anyway, as in this game fighters hyper out when their hull goes critical, making it very very easy to lose a mission because one of the enemy fighters manages to escape.