Samuel Johnson's Dictionary of the English Language (1756) expresses no doubt that the term comes from the word rub:

RUBBER, s. [from rub]

1. One that rubs.

2. The instrument with which one rubs. Swift.

3. A coarse file. Moxon.

4. A game ; a contest ; two games out of three. Collier.

5. A whetstone.

Twenty years earlier, John Kersey, A New English Dictionary: or, A Compleat Collection of the most Proper and Significant Words, and Terms of Art Commonly used in the Language (1739) gives only two rubber-related definitions:

A Rubber, a Rubbing-Cloth.

To play Rubbers, or a double Game at any Sport.

Earlier editions of Kersey from 1706 and 1720 have no entries at all for rubber; and Elisha Coles, An English Dictionary, Explaining the Difficult Terms that are used in Divinity, Husbandry, Physick, Philosophy, Law, Navigation, Mathematicks, and other Arts and Sciences (1717) likewise has no entry for rubber. At least in their early editions the pre-Johnson dictionaries focused on difficult words, and it is highly probable that rubber was widely used in England prior to its appearance in the 1739 Kersey dictionary.

Though both Kersey (1739) and Johnson (1756) offer definitions for rubber in the context of games, neither includes a definition for latex-based rubber nor for anything implying elasticity. This provides circumstantial evidence that rubber as applied to games and sports is not directly connected to what used to be called "India rubber." In fact, the earliest dictionary I have found that mentions "India rubber" is Noah Webster, An American Dictionary of the English Language (1828), where this item appears at the bottom of the entry for rubber:

India rubber, elastic resin, or caoutchouc, a substance produced from the syringe tree of south America; a substance remarkably pliable and elastic.

All of which tends to eliminate the notion of elasticity or bouncing as a possible element the original sense of rubber as used in the context of games—but it doesn't offer much additional insight beyond Johnson's surmise that rubber in games, like the other senses of rubber that he lists, comes from rub.

Two very early instance of rubbers in the context of games appears in Thomas Dekker, "Sloth or The fourth dayes Triumph" in The Seven Deadly Sinnes of London (1606):

Hee [the "Cowardly Generall" Sloth] then gau licenses to all the Vintners, to keep open house, and to emptye their Hogsheades to all commers, who did so, dying their grates into a drunkards blush (to make them knowne from the Grates of a prison) least customers should reele away from them, and hanging out new bushes, that if men at their going out, could not see the signe,yet they might not loose themselues in the bush. He likewise gaue order that dicing-houses, and bowling alleyes should be erected, whereupon a number of poore handy-crafts-men, that beore wrought night and day, made stocks of themselues of ten groates, and crowns a peece, and what by Betting, Lurches, Rubbers and such tricks, they neuer tooke care for a good daies worke afterwards.

and in Thomas Dekker, "Vincents Law" in The Belman of London (1608):

The Dycing ebeator, and the cozening Card-player, walke in the habites of Gentlemen, and cary the faces of honest men. So likewise doe those that are Students in the Vincente Lawe : whose Inne is a Bowling Alley, whose bookes are bowles, and whose law cases are lurches and rubbers. The pastime of bowles is now growne to common exercise, or rather a trade of which some of all companies are frée ; the sport is not so common as the cozenage used in it, which to have it live with credyt and in a good name is called the Vincents Law.

In this Law they which play booty are the Banckers.

He that Betteth is the Gripe.

He that is cozened is the Vincent.

The Gaines gotten is called the Termage.

The Bankers are commonly men apparelled like honest and sub / stanciall Citizens, who come into the Bowling Allies, for a rubbers or so, as though it were rather for sport, then for any gaines, protesting they not whether they win or loose : which carelessnes of theirs is but a shadowe to their pretended knaveries : whilst they are crying Rub, Rub, Rub, and a Great one, In come the spectators dropping one by one, and stand leaning over a Rayle to behold them ; of which oftentimes some simple men that never saw common Bowling Ally before may perhaps be of the number, and is brought in of purpose by one of their owne Brotherhood to be rid of his money : ...

Though Dekker here is describing a crooked game of "bowles," it appear that in honest versions of the game, too, spectators would shout "Rub! Rub! Rub!"—either for encouragement, because "rubbing" (perhaps striking the target pins with the cast bowle) may have been the source of points in the game, or for purposes of calling for wagers. In any event Dekker's description of bowling offers a possible explanation for how rubber (or according to him, "a rubbers") might have emerged in its idiomatic gaming sense.

In explaining how to say "Rubbers at bowls" in Latin, Christopher Wase, Dictionarium Minus: A Compendious Dictionary English-Latin (1662) provides some insight into how the term was understood in English in 1662:

Rubbers (two games) at bowls Gemina in ludo sphærarum victoria.

Rubber match

Posted by Henry on September 11, 2003

In Reply to: Rubber match posted by Bruce Kahl on September 11, 2003

: : : : In sports where 2 wins out of 3 games wins the match, and after 2 games where each team has won one match, the third (and decisive) game is referred to as the "rubber match". Where did the term "rubber march" come from?

: : : There was a discussion under the title of Rubber Match? on 19 August 2003. You can find it stored in the archives.

: : : By the way, a rubber band will play a rubber march.
: : Rub her band and it will turn on her finger. But don't do it in sight of the robber band.

: Ab The Word Detective:

: "According to Paul Dickson's The New Dickson's Baseball Dictionary (Harcourt Brace, 1999), a "rubber game" is "The last and deciding game of a series when the previous games have been split; e.g., the seventh game of the World Series." This tie-breaking sense of "rubber" apparently originated in the pulse-pounding English game of "bowls," or lawn bowling. Despite its name, bowls has little in common with American bowling, and consists of rolling wooden balls (called "bowls") across a level green, the object being to get your ball as close as possible to (but not to hit) a little white ball at the other end of the green. "Rubber" in its tie-breaking sense first appeared in the context of bowls around 1599, and was in use by the card-playing crowd (whist, bridge, etc.) by 1744. A set of three games of bridge is still generally referred to as a "rubber."

: Unfortunately, no one knows where "rubber" in this sense came from. It appears to be unrelated to the elastic sort of "rubber." (Incidentally, our modern elastic "rubber" is short for "India-rubber," from its original source in the East Indies. "Rubber" previously meant anything used to rub, smooth or clean.) Brewer's Dictionary of Phrase and Fable ventures that the term may have referred to two "bowls" rubbing together, a fatal error in the game of bowls. Or it might be a metaphorical use of "rubber" (something that expunges) referring to the "sudden death" third game of a series, the loss of which would conclusively "rub out" the losing team's hopes. But there is, sad to say, no solid evidence for either theory."

I think that 'rubber match' will mean the match that will decide the rubber ie the fifth in a series of five. The term rubber doesn't itself mean decisive.