While I don’t disagree, especially about the New Wave, which I hold responsible for the “death” of sci-fi (at the same time that sci-fi books died because of the trick/unexpected ending, sci-fi movies and games took off because they maintained, for the most part, the positive endings), I submit for your consideration, that genre is about setting and endings/expectations.
She hits the core point again here:
Fantasy outsells sci-fi and I think that’s because readers know what to expect.
And bookended here:
To illustrate how important reader expectations are, I point out Romance. This is big “R” romance, where the ending is prescribed. That ending is HEA (Happily Ever After) or HEAFN (HEA for now). Readers want to know what they’re getting.
I submit that readers want positive endings, heroic characters, the good guys winning, the bad guys losing. No whiny, depressed losers emoting all over the page. I want to see sci-fi get back to that.
This is not the first time I've heard such observations. Many years ago, Mike Pondsmith wrote this into the Game Master advice section of his Mekton Zeta tabltop role-playing game. He cited (then) well-known examples of fulfilling audience expectations, but not necessarily in the straight-forward manner initially put to the audience. He wrote similar advice for his Cyberpunk 2020 supplement, Listen Up You Primitive Screwheads.
The point here is expectation management. People group things they find to be alike and label them for ease of reference; that's what genres are- groups of stories that deliver similar experiences to an audience. People like familiarity because it's reliable; you know what you're going to get. So, if you want to succeed, you need to deliver on the expectations of that audience. Danielle Steel didn't goat-fuck her readers into a multi-media empire of romance, and neither will you. (You'd think the Big 5's current regime would know this, but we know better; they don't.)
However, fulfilling those expectations need not--and some say should not--be as straight-forward as a drag race. Going back to Pondsmith, his example of making good on expectations in an unexpected manner was the original Super-Dimensional Fortress Macross; the promise was that we Earthlings would defeat this hostile alien invader and preserve Mankind thereafter. The swerve was that a good chunk of Earth got blasted and we ended up having to integrate with their remnants after the war. Sure, Mankind won, but not without sacrifice and not without complication after the fact.
See what I'm after here? There's your liminal space, where you can work your magic and define yourself as a writer. Scott Lynch broke out as strong as he did because his debut novel did just that, on multiple levels; as the protagonist Locke Lamora schemed and scammed his way towards fulfilling your expectations of the fantasy caper he set up early on, so did he pull a swerve on you and got you looking away while he pulled a con on those expectations so that what he delivered when he delivered turned out to be not how you got what you wanted, but you got it nonetheless.
Robert Howard's Conan went on adventures, or had them thrust upon him, that fit a general plot profile; it was how Howard executed it that made him stand out as a writer, and in seeing the difference between two different characters in the same scenario (Conan: The Phoenix on the Sword; Kull: By This Axe I Rule) you can observe how Howard's choices differed even when he adapted the latter into the former.
Now, accomplished authors already know this; they can tell you how they made this happen once they put this knowledge to practical use. But you? You're new, or you're struggling and looking for insight, or otherwise dealing with issues in developing your craft or story (or both). Yes, knowing the structure of narrative is important, but being lazy or incompetent about it means becoming like the folks I rant about over at my main blog and that's no good for you personally, professionally, or artistically. You need to master the tools, not to be a tool.
There is no shortcut to be had here. You have to get it before you can make it. Then you have to get it right, and in time you'll become so good that you can't get it wrong. There's no other way to mastery than to make the work, so go on and get on with making your 10,000 pots; the sooner you develop-by-doing, the sooner you will become wise and skilled enough to translate what your mind imagines into stories that your audience cannot wait to pay you for.
So learn what your audience expects from the story you're writing. Then learn how to deliver on those expectations. Then learn how to make use of the room you have to satisfy without failing that fulfillment promise. Remember: Anakin Skywalker did bring balance to The Force. It's how he did it that makes George Lucas rightly revered, not that he did it at all.