The GOP has some choices.

1) Give up the expectation of winning elections at the national level.
2) Grow the sizes of the demographic groups that currently support them.
3) Increase turnout of those supporters (without pissing off current supporters enough to turn them to
non-supporters in larger numbers than the marginal turnout gain).
4) Persuade more people in growing demographic groups to agree to support GOP candidates, by...
4.1) Rational appeals to accept the current GOP policy positions
4.2) Irrational appeals...
4.2.1) to accept the current GOP policy positions (again, without pissing off current supporters enough to turn them to non-supporters in larger numbers than the marginal turnout gain).
4.2.2) based on running more "token" candidates from the growing minorities (once more, without pissing off current supporters enough to turn them to non-supporters in larger numbers than the marginal turnout gain).
5) Shrink the number of active voters in demographic groups that oppose them by...
5.1) Suppressing the fraction voting
5.2) Reduce the population of those demographic groups
6) Change some of the policy positions to appeal to more people, particularly in growing demographic groups.

#1 is implausible.

#2 won't help any time soon as well as being really difficult. #3 is difficult, but there might be some tactics they can copy from the Obama team to get the GOP a little more. The obsessively empirical mindset that drove the Obama team, however, seems likely problematic given the GOP's apparent tendency to refuse acceptance to inconvenient empirical data.

#4.1 may not even be a self-consistent option, given the current attitudes apparently in vogue in the GOP base about intellectuals. The current "leading lights" seem at best pale shadows of William Buckley or Ayn Rand, who themselves were hardly titanic luminaries relative to the wider arena of politics. #4.2.1 might have some potentials I don't see (irrational persuasion is not my forte), but it looks like they're at the limits where increased scaremongering will drive away more moderates than it rallies terrified protofascists. #4.2.2 conceivably might work, except that such tokens are already hard to find without considerable slacking on ideological purity.

5.1 is criminal, but if the courts don't respond sharply the approach has significant risk of violent backlash. 5.2 verges on genocidal. Given US history, I don't put either out of the question. A replay of 1960s civil rights violence is about the best scenario to expect if they go that route.

That leaves #6. Unfortunately, that's going to involve pissing off some of their current core base, in a big way. The best chance that I see looks to be for the Moneycon and Jingocon factions to dig in and split off from the Theocon/Xenocon Tea Party types -- which will lose them at least half their numbers, and thus possibly the GOP/Republican name. They can then aim for a more Libertarian shore, and hope to pull in those plus some of the DIY democrats. Push against any legislative effort on the the abortion and gay marriage issues (voting "abstain" or "present" or not at all, rather than "yea" or "nay"), but once anything passes or is ruled on by the courts, accept it as the new status quo to be ignored once more. Recognize excessive corporate power is potentially as toxic as excess government power. Aim for "smaller government", but figure out where society will have a safety net and how it will maintain social mobility. They'll probably need to push for a change to the voting systems (concordet, instant run-off, whatever) to allow third party viability, as a way of buying some support from the Tea Party. (The democrats may prefer the GOP self-destruct, but might be long-sighted enough to see their own potential for fission.) That might be combined by addressing the problem of redistricting, and coming up with some principle yielding a long-term stable algorithm to take that source of political shenanigans off the table as the electoral college algorithm did for that back in the 1950s or so... though finally ditching that antique might also fit in well, and be needed if they don't want to take approach #1 on the presidency.

The biggest potential obstacle would be that I'm not sure the current officeholders are sufficiently representative of the GOP's internal factions to allow such a split to be graceful. Also, I may well be deluding myself on the coherence of such a platform block.