Expected accuracy from Monte Carlo simulations?

[Norm]

Eric, I really do not think you read the two threads carefully enough. Yes, a couple of his posts said he welcomed input. The remainder of his 69 posts refused input, were belligerent, denied the importance of statistics, threw insults in every direction, claimed it was legal to use the device at the table (then admitted it wasn’t and denied he said it – then reversed again claiming it to be legal), and consistently claimed other posts said things they didn’t. He then ignored instructions to stay out of the General Forum until he investigated Standard Error.

I would also like to hear what you think was embarrassing about discussions of cloud-computing, RNGs, and multi-core speedups. I have spent rather a large amount of time investigating these areas.

This post seems out of character. Perhaps you just got up on the wrong side of the bed this morning.

[ericfarmer]

Eric, I really do not think you read the two threads carefully enough. Yes, a couple of his posts said he welcomed input. The remainder of his 69 posts refused input, were belligerent, denied the importance of statistics, threw insults in every direction, claimed it was legal to use the device at the table (then admitted it wasn’t and denied he said it – then reversed again claiming it to be legal), and consistently claimed other posts said things they didn’t. He then ignored instructions to stay out of the General Forum until he investigated Standard Error.

I would also like to hear what you think was embarrassing about discussions of cloud-computing, RNGs, and multi-core speedups. I have spent rather a large amount of time investigating these areas.

This post seems out of character. Perhaps you just got up on the wrong side of the bed this morning.

I apologize if my initial post was snarky or grumpy-- my intent was not to add to the grumpiness, but to point out that the community reaction seemed pretty grumpy from the start, and to suggest that we try to actually *quantify* the discussion, ask for some data, etc., rather than engage in generalizations.

I guess I didn't read the conversation the way everyone else has; I found the tone of many of the responses to be contrarian from the start. Yes, tomker eventually got frustrated, but I think his frustration was/is deserved. Indeed, here is your second response to the initial post (the first being a brief comment about representative subsets):

1. "Cloud" is an advertising term, not a technology term. It has no specific technology meaning. It means whatever the advertiser claims it means. Kind of like "organic".
2. “Cloud” services nearly always use very slow processors, like the AMD sixteen core chips.
3. Representative decks generally means multiple representative decks, which your description appears to match. It is a shortcut used by “simulators” and does not present good results. I tried this technique a couple decades ago. I decided that it was not accurate. Indeed, there really is no reason that it should be considered accurate.
4. You claim to have used a DigitalOcean 16 core PC. I hope you realize that you are only using one core, and that core is very slow as 16 core PCs have slow cores.
5. A few hundred thousand hands as a sample is completely, absolutely useless. With eight million hands, the standard error would be 0.13. What do you think it is with a few hundred thousand? Even if you did a REAL simulation, instead of using an invalid technique -- in a few hundred thousand hand sim, you can actually prove that basic strategy in an H17 game has positive expectation.

Sorry if this seems harsh. But, you need to do some reading.

It is harsh:

2. I think you are saying that the individual processors on a 12-, 16-, 20-core, etc. machine tend to be slower than, say, a 3+ GHz 2- or 4-processor workstation? Fair enough... but so what, when there are 16 (or however many) of them? As I read it, tomker's task was indeed embarrassingly parallel (pre-computing a bunch of stored basic strategy tables), perfect for throwing a lot of cores at. You make a good point later on that the actual *speedup* from using n cores over one is not actually n-fold due to contention (the extent of which varies significantly across different applications), but that doesn't make parallelizing not still useful in this case. This is nothing new; in my past analyses, e.g. evaluating "actual" PE for indexed strategies, I did exactly the same thing tomker describes.

4. It's not clear to me how anyone inferred that he was "only using one core," when he never said that.

5. Having gone back and read in more detail, I see now that tomker already provided a specific answer to my initial question (see here): he is observing expected values that agree "to at least 1 cent per dollar." As Nyne's helpful post shows, this is a reasonable, if somewhat vague, characterization of what he would observe with 100,000 sample rounds. That is, assuming an *actual* standard deviation of sigma=1.1417 (as a fraction of initial wager), the estimated expected return using 100,000 samples would be within 0.01 (i.e., 1% of initial wager) over 99% of the time. (Tomker, can you verify that I'm re-stating your observations correctly?)

Granted, this is arguably "useless" as you put it, Norm, since strategy decisions often hinge on differences in expected return that are much less than this. And if that's all he is trying to do (justify the accuracy of strategy suggestions), well, then instead of dismissing him altogether, let's explain *why* he needs more accuracy... and how *much* more sounds like an actually interesting question: how few samples do you need merely to accurately estimate the *order statistics* of the EVs of the various playing strategy options (stand, hit, etc.) with reasonable confidence?

[Three]

Tomker, Eric is one of the software experts I spoke of that can be most helpful to you. He is very impartial in any analysis. He just is interested in getting answers right to get at the truth of any matter. If you think people aren't giving your app a fair shake or are incorrect in their analysis, open up the app to Eric and respect whatever he comes up with. It will be an unbiased analysis you can trust is true. I am often in the minority camp on issues that have no right or wrong, just benefits and liabilities. Quantifying the same can be difficult and data can be cherry picked to get the results any particular person wants. Eric's results will be as unbiased as mathematically possible for the analyzation technique he uses. I was hoping he would tune into the conversation. Thanks Eric I am interested in what you think. I too saw the potential for interesting output here.

Hopefully you can help him to make sure he maximizes the potential for what he wants to do.

[Norm]

Eric, tomker was treated vastly more politely than he would have been in other BJ forums, particularly since he came here to sell an app that does not provide accurate results. Indeed, your post was the harshest of all the posts in these two threads. What is “frustrating” is dealing with numerous people wandering in to sell apps or systems. Such is the life on a forum. We ban outright about 350 new members a day. Over 100,000 a year. I decided to let this one remain as perhaps it could spur a decent discussion. But, instead of listening, he had a somewhat paranoid reaction. When we tried to explain his standard error was too high, he responded that he didn’t know what SE was, and later that knowledge of stats was not necessary. Several members tried to explain SE to him – but he showed no interest at all. That’s when he completely lost us. If you show little interest in learning, repeatedly claim others are calling you a liar (as opposed to incorrect), and suggest that basic statistics are not needed; you are going to have difficulty selling stats on a forum.

One point on DigitalOcean. Their advertising states that you are getting a 16-core machine in one of their plans. Nowhere does it say that they are virtual cores shared with other users. Potential and current customers have been asking them for over a year for the physical to virtual ratio, and they have been silent. That’s the problem that I have with the use of the word “cloud”. It means what the advertiser wants it to mean – and that does bug me.

BTW, glad to see you posting again.

[ericfarmer]

Eric, tomker was treated vastly more politely than he would have been in other BJ forums, particularly since he came here to sell an app that does not provide accurate results. Indeed, your post was the harshest of all the posts in these two threads. What is “frustrating” is dealing with numerous people wandering in to sell apps or systems.

I think I have come across as "attacking" this morning. I certainly don't mean to do so-- but at the same time, I struggle probably more than I should with leaving be claims that are inaccurate, misleading, etc... especially when such claims are part of a larger accusation of someone else *themselves* being inaccurate, misleading, etc.! I'm sorry if it sounds like I'm attacking you, but if you say something incorrect, I can't keep my mouth shut .

So, let's discuss a specific example here. I can't actually respond to the comment directly, since the thread has been closed.

Briefly, let's flip a fair coin 100,000 times, and record the fraction X of those times that we observe heads. Now let's perform this 100,000-toss experiment 10 times, recording the fractions X_1, X_2, ..., X_10.

The linked comment contains a bar chart showing a particular observation of the order statistics of |X_i - 1/2|. Norm, you point out that "we can see an apparent bunching of results. There is no reason for such a pattern with a simple coin-toss and 100k hands, suggesting that it is an artifact of the RNG." I assume by this "bunching" you mean the "tiers," or groups of three-ish samples all with approximately the same deviation from 1/2.

I feel like you have a habit of picking on RNGs . The linked chart (as well as the subsequent larger sample provided by tomker) is a perfectly reasonable observation, and the "bunching" has nothing to do with the quality of the underlying RNG. This is *expected* behavior, the likelihood of which can be computed via relatively straightforward means. Quantifying this likelihood is difficult, since I'm not sure exactly what "event" it is that you are claiming is unlikely. For example, is it just that there is a large "jump" from the 6th order statistic to the 7th, or that there are multiple such "large" jumps, etc.?

Until we can characterize exactly what appears to be biased/correlated about this sample, let's try looking at it another way. In these two images (click on "first image" or "second image"), I have generated 2x9 similar samples, plotted in the same way as described above. One group of 9 was generated using a "low-quality" linear congruential generator, while the other was generated using the Mersenne Twister. Can you tell which is which? (Note the occurrence of "bunching/tiers" in both cases.)

[tomker]

... Tomker said his standard error was .1% in a previous post rather than Nyne's .361% for the sample size of his Monte Carlo simulations. ...

Absolutely incorrect and I challenge you to find any post I made anywhere that says this.

In your haste to make it look like I'm wrong or my software is bad, you've now resorted to simply making up these critical numbers that we're having multi-page arguments about.

[tomker]

...
But back on subject: Nyne's example calculation is a good one, and your response suggests that perhaps we are just confusing units, i.e. *fractions* of initial wager, versus *percent* of initial wager? In other words, when you say "accurate to within 0.01," do you mean within a dime of a $10 wager, or within a tenth of a penny?

Also, I can't seem to find the graph in a different thread that you mention (unless it's the coin-tossing example). Can you provide a link?
...

Hi Eric, nice to meet you and thanks for your interest!

All of the EVs I've seen online (and some that have been discussed in these threads) have been like "doubling has an EV of 0.178" which apparently assumes an initial bet of 1.0. In my app I have reproduced these numbers and when I say they're accurate to 0.01, that's what I'm referring to, i.e., the "0.17" part is accurate. Hope this clears things up!

I made a post in the previous thread saying that if you purchase the "Ultimate version" upgrade in my app ($6) and email me your iTunes receipt, I will be happy to refund you the entire purchase price, plus tax, via Paypal. Unfortunately Apple has not made a system where I can gift the upgrade to people so this is the best I can do.

Edit: whoops, forgot to attach the graph! If you have any questions about how I made it, I'd be happy to answer them!

[tomker]

... claimed it was legal to use the device at the table (then admitted it wasn’t and denied he said it ...

Happily this is a message board and not a phone conversation that you forgot to record, Norm. If you think I said something then you can just quote it.

I admit that I told a story that was a bit ambiguous and possibly confusing but I never used a device at a table, never claimed it was legal, and in fact my app has a section that explicitly says that it's not allowed.

You are now behaving like Tthree, simply making up things that I never said and then arguing against them.

[Norm]

Erik, I do indeed pick on poor RNGs. There was a package called BJ678 that was sold for years in this field. After someone gave a result after a 20,000,000 hand sim, it was pointed out that that wasn’t enough hands. He said that running anything over 20M hands with that software always gave the same result as 20M hands -- so he stopped running more hands. This is a clear problem with the software, almost definitely with the RNG. I’ve seen posts using the Excel RNG, which was known to have a period of about 30,000. It is also known that RNGs, even with very long periods, can cause aberrations in results that show up only in certain models. The point of the animation that I posted was to show that dispersion may look good when you look at it one way – but not another.

APs often depend on stats for their livelihood. I’ll continue to make posts about poor RNGs.

On tomker’s attitude, tthree has been very polite and encouraging to him. Look at tomker’s post a few minutes ago accusing tthree of purposely trying to make his software look bad and making up numbers. Tom falsely accused me of calling him a liar three times when I merely said he was incorrect. He now falsely accuses tthree. His attitude does not engender helpful attitudes in others. He is asking us to perform work for him, and responds with accusations and snark.

[tomker]

... That is, assuming an *actual* standard deviation of sigma=1.1417 (as a fraction of initial wager), the estimated expected return using 100,000 samples would be within 0.01 (i.e., 1% of initial wager) over 99% of the time. (Tomker, can you verify that I'm re-stating your observations correctly?) ...

Yes, exactly. Actually I saw 97% with the samples I used to make the graph and not 99% but assuming I had ~3 slightly unlucky samples, my observed results match this prediction exactly.

[tomker]

...
Until we can characterize exactly what appears to be biased/correlated about this sample, let's try looking at it another way. In these two images (click on "first image" or "second image"), I have generated 2x9 similar samples, plotted in the same way as described above. One group of 9 was generated using a "low-quality" linear congruential generator, while the other was generated using the Mersenne Twister. Can you tell which is which? (Note the occurrence of "bunching/tiers" in both cases.)

Actually it's a little bit funnier than that. In his haste to try to make me look wrong he posted a graph from his "good RNG" where almost all of the values are high and the overall graph shows basically the opposite curve vs. what is expected.

I'm not saying his RNG is bad but clearly a sample size of 10 is not enough to determine anything.

I have superimposed Norm's graph over my "higher resolution" graph.
wrongshape.png

[Norm]

Happily this is a message board and not a phone conversation that you forgot to record, Norm. If you think I said something then you can just quote it.

I admit that I told a story that was a bit ambiguous and possibly confusing but I never used a device at a table, never claimed it was legal, and in fact my app has a section that explicitly says that it's not allowed.

You are now behaving like Tthree, simply making up things that I never said and then arguing against them.

OK, here are two of your quotes:

BTW... looked up NRS 465.075.

I think it's pretty obvious that the law is referring to calculating probabilities for a specific instance of blackjack and not blackjack in general.

Otherwise, it would be illegal to refer to one of those little plastic cards with basic strategy printed on it, and people do that all the time.

You told me that using my app at a casino is illegal but you clearly didn't read the law in question and your only evidence that it's illegal is that a different app that does a different thing is illegal.

By these quotes, you are suggesting that your app is legal in a casino, just like looking at a plastic card. You went on to say I didn't read the law, and my only evidence is that some other app is illegal. No, I am quite familiar with the law, have discussed it with a gaming attorney, and have it posted on three sites.

Finally, if you make another false accusation against tthree, you will be suspended. NO ONE here has suggested, even slightly, that you have any nefarious purpose in your posts or software. You must stop reading things into posts that aren’t there and treat members with respect.

[tomker]

... how few samples do you need merely to accurately estimate the *order statistics* of the EVs of the various playing strategy options (stand, hit, etc.) with reasonable confidence?

BTW -- I agree that this is interesting and towards the end of the last thread I did an experiment that compared the order of 100k EVs vs. 10M EVs. Unfortunately I only played through 200 decisions but at no point did the 100k EVs indicate a different action than the 10M EVs.

If I get some time today, or in the next couple of days, I'd like to go through the basic strategy chart and calculate the EV deltas between 1st and 2nd best actions. Then I'd like to use a Monte Carlo simulation to generate a "heat map" of how likely each scenario is to be encountered.

Using those two charts and some multiplication it should be very easy to quantify the value of different levels of EV accuracy.