See the top rated post in this thread. Click here

Page 1 of 3 123 LastLast
Results 1 to 13 of 36

Thread: Statistical Significance

  1. #1


    6 out of 6 members found this post helpful. Did you find this post helpful? Yes | No

    Statistical Significance

    There appears to be a lot of confusion about and misunderstanding of the meaning of statistical significance in some recent threads when discussing the value of simulated results vs the results of actual play.

    The software and math guys (me among them) are correctly saying that, when simulating, we sometimes need to simulate billions of rounds in order to arrive at statistically valid numerical results. At the same time, some of those trying to evaluate the results of their actual play for the purposes of finding holes in their game or deciding whether to switch systems, are throwing up their hands and saying, "What's the point? I'll never play billions of rounds within my lifetime?"

    Is there a contradiction here? If not, then how can the two worlds be reconciled? This will be my attempt to try and clear things up.

    The main concept to grasp is that all observed results are statistically significant to within some margin of error. The more samples you observe, the smaller that margin of error becomes. Obviously, below some threshold, the number of samples can be insignificant in both practical and mathematical terms. You may have seen this referred to as the Standard Error.

    Sometimes concepts like this are easiest to grasp when considering the ridiculous extremes. And I do mean ridiculous! For example, it should be easy to see that for flipping a coin once, the observed result will either be 100% heads or 100% tails and so it will differ from the known result of 50% by 50%. Now if we imagine being able to toss that coin an infinite number of times, then the result will become infinitely close to 50% and the standard error will become infinitely close to zero. Notice that I didn't say that the result will become 50% and the standard error will become zero, but they will become close to within some minuscule range (infinitely close to zero) with some high probability (infinitely close to 100%)

    Of course, we have no use for these extreme results. We live in the finite world. So how many samples is enough? Well, it depends on what you are observing and what you want to use the results for.

    For a simple process like tossing a coin, it turns out that the standard error is 0.5/sqrt(samples), which converges fairly quickly. After only 10,000 tosses, the standard error is 0.005 or 0.5%, which means that you have a 99.7% chance the result you have observed is within +/- 3 standard errors, or within +/-1.5% of the true result. That's a 3% margin of error. Good enough for you? Maybe (but I hope not). Good enough for a simulation who's goal is to determine the true result to within 2 decimal places? Absolutely not.

    For blackjack, a typically used standard deviation for the EV of a single round is 1.1. So the standard error is 1.1/sqrt(rounds). After 10,000 rounds you have a 99.7% chance of being able to calculate the true EV to within +/-3.3%. That's a 6.6% margin of error!! After a million rounds you're down to a 0.66% margin of error. Maybe good enough for you to estimate your expected win rate to within a few dollars. Certainly not small enough to be able to declare that system X has a 0.57% EV and system Y has a 0.64% EV and therefore system Y is superior, and that's after 1 million rounds.

    And that's the point of it. We use simulation as a method of calculating specific numbers which have true (unknown) values, but which are too difficult to calculate directly, and we want those numbers to be within a certain level of accuracy. The more rounds we simulate, the closer our numbers will be to the true (unknown) result. We can then use those numbers for making other calculations or for comparing systems. Some simulations, like the ones done in order to compute SCORE are accumulating many different statistics, some of which are for events which are more rare than others and so billions of iterations are needed in order to reduce the standard error for those rare events to an acceptable size.

    Now, does this mean you need to play billions of live rounds in order to benefit from the knowledge obtained via the simulation? The answer is "No". Unlike the simulator, your goal is not to achieve the precise statistical result that the game offers. Your goal is simply to extract the money at a rate close to that predicted by the simulator. If your actual results are different by a few decimal points, then you will still be making money. If fact, if your results are within one standard deviation of the predicted results after N0 rounds, then you will still be making money. This level of accuracy is attainable by playing a number of rounds which is certainly achievable within your playing years.

    In summary:
    • Multi-billion round simulations are needed in order to get the precise statistically significant numbers you need to make informed decisions about your play. If you are making decisions based on the results of short live play experiments of 10,000 rounds, then you are making a mistake. There is a significant chance that the inferior system could out perform the superior one over the course of the experiment due to excessive margin for error. This is especially true of you are only making tweaks to an existing system, as opposed to comparing different systems.
    • You don't need to achieve the precise results predicted by the simulator. You only need to achieve results which are somewhat close in order to get the money. This can be done within a much smaller number of rounds played which is easily attainable.
    • When comparing systems and other decisions, use the simulation results to make the decision. These will tell you which has the higher potential. If the difference in potential is large enough, then you can play enough rounds to enjoy the benefit.


    I hope this helps!

  2. #2
    Senior Member Bodarc's Avatar
    Join Date
    Aug 2013
    Location
    136 miles North of West
    Posts
    1,949


    Did you find this post helpful? Yes | No
    Thanks Gronbog
    Play within your bankroll, pick your games with care and learn everything you can about the game. The winning will come. It has to. It's in the cards. -- Bryce Carlson

  3. #3


    0 out of 3 members found this post helpful. Did you find this post helpful? Yes | No
    No one was arguing any of that, at least I wasnt. I know there's a difference between knowing if something is statistically significant by simulating a billion rounds and knowing you dont need a billion rounds to reach that expected result. Of course you dont need a billion rounds to reach your expected target. My point is, these people advocating side counts are completely delusional, they think their system is some type of godly system where in fact they're just hitting positive variance just like they would with any other count. Side counts in a shoe game? There just isnt a high enough frequency sample to have a surplus or deficit of any card value to make side counting that big of a benefit in the long run. But of course they use their anecdotal data and claim they're right without proving to us with a billion round simulation because they also claim there's no simulator that can do what their super non-linear system does LOL. Instead they go side count 100's of cards and hit their EV in 10 minutes with their SCORE OF 500.

    I just wish I could do a case study of just playing HiLo with full indices while occasionally counting multiple tables simultaneously when the opportunity arises VS = their super godly count only counting one table at all times. We would play only SHOE games for 1800 hours a year(36 hours a week) and see who comes out on top. Everything stays constant. We play same games, same penetration, same everything and we'll see who comes out on top. I would blow them out of the water. Of course one of the super side counters might claim why does it have to be mutually exclusive, why cant the super side counter also count two tables at once? And my answer will be, I would like to see them try LOL.
    Last edited by LoneWoLF; 06-29-2016 at 03:34 PM.

  4. #4
    Random number herder Norm's Avatar
    Join Date
    Dec 2011
    Location
    The mote in God's eye
    Posts
    12,461
    Blog Entries
    59


    Did you find this post helpful? Yes | No
    Quote Originally Posted by LoneWoLF View Post
    My point is, these people advocating side counts are completely delusional
    You seem to be lumping a huge number of people using numerous different methodologies into one group. Then, you come to a conclusion that you think covers all of them.
    "I don't think outside the box; I think of what I can do with the box." - Henri Matisse

  5. #5


    Did you find this post helpful? Yes | No
    So, mathematically, if you knew the number of people who attempted to be a professional AP BJ, player, the number that succeeded and were still earning their living playing BJ after 15 years, after 10 years, after 5 years, after 1 year, we would have sufficient valid statistical information to make a career decision.

  6. #6


    Did you find this post helpful? Yes | No
    Quote Originally Posted by Gronbog View Post
    There appears to be a lot of confusion about and misunderstanding of the meaning of statistical significance in some recent threads when discussing the value of simulated results vs the results of actual play.

    The software and math guys (me among them) are correctly saying that, when simulating, we sometimes need to simulate billions of rounds in order to arrive at statistically valid numerical results. At the same time, some of those trying to evaluate the results of their actual play for the purposes of finding holes in their game or deciding whether to switch systems, are throwing up their hands and saying, "What's the point? I'll never play billions of rounds within my lifetime?"

    Is there a contradiction here? If not, then how can the two worlds be reconciled? This will be my attempt to try and clear things up.

    The main concept to grasp is that all observed results are statistically significant to within some margin of error. The more samples you observe, the smaller that margin of error becomes. Obviously, below some threshold, the number of samples can be insignificant in both practical and mathematical terms. You may have seen this referred to as the Standard Error.

    Sometimes concepts like this are easiest to grasp when considering the ridiculous extremes. And I do mean ridiculous! For example, it should be easy to see that for flipping a coin once, the observed result will either be 100% heads or 100% tails and so it will differ from the known result of 50% by 50%. Now if we imagine being able to toss that coin an infinite number of times, then the result will become infinitely close to 50% and the standard error will become infinitely close to zero. Notice that I didn't say that the result will become 50% and the standard error will become zero, but they will become close to within some minuscule range (infinitely close to zero) with some high probability (infinitely close to 100%)

    Of course, we have no use for these extreme results. We live in the finite world. So how many samples is enough? Well, it depends on what you are observing and what you want to use the results for.

    For a simple process like tossing a coin, it turns out that the standard error is 0.5/sqrt(samples), which converges fairly quickly. After only 10,000 tosses, the standard error is 0.005 or 0.5%, which means that you have a 99.7% chance the result you have observed is within +/- 3 standard errors, or within +/-1.5% of the true result. That's a 3% margin of error. Good enough for you? Maybe (but I hope not). Good enough for a simulation who's goal is to determine the true result to within 2 decimal places? Absolutely not.

    For blackjack, a typically used standard deviation for the EV of a single round is 1.1. So the standard error is 1.1/sqrt(rounds). After 10,000 rounds you have a 99.7% chance of being able to calculate the true EV to within +/-3.3%. That's a 6.6% margin of error!! After a million rounds you're down to a 0.66% margin of error. Maybe good enough for you to estimate your expected win rate to within a few dollars. Certainly not small enough to be able to declare that system X has a 0.57% EV and system Y has a 0.64% EV and therefore system Y is superior, and that's after 1 million rounds.

    And that's the point of it. We use simulation as a method of calculating specific numbers which have true (unknown) values, but which are too difficult to calculate directly, and we want those numbers to be within a certain level of accuracy. The more rounds we simulate, the closer our numbers will be to the true (unknown) result. We can then use those numbers for making other calculations or for comparing systems. Some simulations, like the ones done in order to compute SCORE are accumulating many different statistics, some of which are for events which are more rare than others and so billions of iterations are needed in order to reduce the standard error for those rare events to an acceptable size.

    Now, does this mean you need to play billions of live rounds in order to benefit from the knowledge obtained via the simulation? The answer is "No". Unlike the simulator, your goal is not to achieve the precise statistical result that the game offers. Your goal is simply to extract the money at a rate close to that predicted by the simulator. If your actual results are different by a few decimal points, then you will still be making money. If fact, if your results are within one standard deviation of the predicted results after N0 rounds, then you will still be making money. This level of accuracy is attainable by playing a number of rounds which is certainly achievable within your playing years.

    In summary:
    • Multi-billion round simulations are needed in order to get the precise statistically significant numbers you need to make informed decisions about your play. If you are making decisions based on the results of short live play experiments of 10,000 rounds, then you are making a mistake. There is a significant chance that the inferior system could out perform the superior one over the course of the experiment due to excessive margin for error. This is especially true of you are only making tweaks to an existing system, as opposed to comparing different systems.
    • You don't need to achieve the precise results predicted by the simulator. You only need to achieve results which are somewhat close in order to get the money. This can be done within a much smaller number of rounds played which is easily attainable.
    • When comparing systems and other decisions, use the simulation results to make the decision. These will tell you which has the higher potential. If the difference in potential is large enough, then you can play enough rounds to enjoy the benefit.


    I hope this helps!
    Sure does help!! Thanks.

  7. #7
    Senior Member
    Join Date
    Dec 2011
    Location
    3rd rock from Sol, Milky Way Galaxy
    Posts
    14,158


    Did you find this post helpful? Yes | No
    Quote Originally Posted by LoneWoLF View Post
    There just isnt a high enough frequency sample to have a surplus or deficit of any card value to make side counting that big of a benefit in the long run. But of course they use their anecdotal data and claim they're right without proving to us with a billion round simulation because they also claim there's no simulator that can do what their super non-linear system does
    Wow. You just can't learn anything. I explained the error of this thought quite well TWICE the last time you said it. Norm has a nice set of graphs of the EV for the various decision options for each matchup he found interesting:

    https://www.card-counting.com/cvcxonlineviewer3.htm

    Now if you were to redo the graph of a poorly correlated play using the side count(s) that are strong cards concerning the play your correlation if tags would go up. The graph would show a fast gain in EV after the index is exceeded rather than an anemic gain. This is because each increment in TC represents a bin full of equivalently rated situations. One has poorly correlated bins so the bins average shows little gain. The other has highly correlated bins so the average of each successive bin shows a lot of gain in EV. Some of the bin might be populated by deficit side counted cards with a strong playing count TC while others in the bin have a normal amount of side counted cards and others have a surplus and a low playing TC. All these situations are equated as their average when determining the EV of the decision for that bin. The guy side counting and the guy not side counting might both be in the first bin above the index and the side counted cards are at expectation BUT the guy side counting has a much larger EV with this hand because of the higher correlation of all the other situations that populate the bin. The guy straight counting would have almost no EV in the deviation because his bin is so poorly correlated to the decision. His EV is the average of all the situations that populate the bin. Their combined EV is almost nothing for making the deviation.

    Like I have always said there is value in knowing side counted cards are at expectation. The value isn't about the current deck composition because neither really has much of a handle on that. The value is in the much higher increase in EV as the index is exceeded due to the average of all the situations that populate each successive bin having a much higher correlation to the decision.

    The I18 shows the most important plays for your count. If you add side counting many plays that are worthless become more valuable than some in the I18. Some decisions will pretty much be just based on the side count. I am not saying you should side count shoe games. Only your understanding of what is going on is flawed. All those that use ace neutral counts side count shoe games. Side counting the ace can be worth up to 20% gain in EV in shoe games when used with an ace neutral main count. You have gain from increased correlation of almost every matchup. You have gain in betting more because of the higher EV and reduced variance of the decision. There doesn't have to be a surplus or deficit of aces to realize these huge gains.

  8. #8


    Did you find this post helpful? Yes | No
    Quote Originally Posted by Tthree View Post
    Wow. You just can't learn anything. I explained the error of this thought quite well TWICE the last time you said it. Norm has a nice set of graphs of the EV for the various decision options for each matchup he found interesting:

    https://www.card-counting.com/cvcxonlineviewer3.htm

    Now if you were to redo the graph of a poorly correlated play using the side count(s) that are strong cards concerning the play your correlation if tags would go up. The graph would show a fast gain in EV after the index is exceeded rather than an anemic gain. This is because each increment in TC represents a bin full of equivalently rated situations. One has poorly correlated bins so the bins average shows little gain. The other has highly correlated bins so the average of each successive bin shows a lot of gain in EV. Some of the bin might be populated by deficit side counted cards with a strong playing count TC while others in the bin have a normal amount of side counted cards and others have a surplus and a low playing TC. All these situations are equated as their average when determining the EV of the decision for that bin. The guy side counting and the guy not side counting might both be in the first bin above the index and the side counted cards are at expectation BUT the guy side counting has a much larger EV with this hand because of the higher correlation of all the other situations that populate the bin. The guy straight counting would have almost no EV in the deviation because his bin is so poorly correlated to the decision. His EV is the average of all the situations that populate the bin. Their combined EV is almost nothing for making the deviation.

    Like I have always said there is value in knowing side counted cards are at expectation. The value isn't about the current deck composition because neither really has much of a handle on that. The value is in the much higher increase in EV as the index is exceeded due to the average of all the situations that populate each successive bin having a much higher correlation to the decision.

    The I18 shows the most important plays for your count. If you add side counting many plays that are worthless become more valuable than some in the I18. Some decisions will pretty much be just based on the side count. I am not saying you should side count shoe games. Only your understanding of what is going on is flawed. All those that use ace neutral counts side count shoe games. Side counting the ace can be worth up to 20% gain in EV in shoe games when used with an ace neutral main count. You have gain from increased correlation of almost every matchup. You have gain in betting more because of the higher EV and reduced variance of the decision. There doesn't have to be a surplus or deficit of aces to realize these huge gains.
    Irrelevant

  9. #9


    Did you find this post helpful? Yes | No
    Quote Originally Posted by Norm View Post
    You seem to be lumping a huge number of people using numerous different methodologies into one group. Then, you come to a conclusion that you think covers all of them.
    Funny you say that, cause the other side of the coin is the same way. The side counters clump all simple practitioners in the same class, but you dont say anything about them doing that.

  10. #10
    Senior Member
    Join Date
    Dec 2011
    Location
    3rd rock from Sol, Milky Way Galaxy
    Posts
    14,158


    Did you find this post helpful? Yes | No
    Great post Gron. Since I was talking about playing decisions in my last post. We consider 1,000,000 data points to be enough for significant results but for playing decisions that depends heavily on the correlation of the count to the EoRs of the play. For strongly correlated plays don't need as much data to converge and poorly correlated plays may need far more than 1,000,000 data points to even start to converge. There is no set number for significance but the way the results behave show you when the are becoming predictable and when they are randomly scattered. I like a graphical representations like Norm's in the previous post's link. Then I can use the raw data as backup for getting specific.

  11. #11
    Random number herder Norm's Avatar
    Join Date
    Dec 2011
    Location
    The mote in God's eye
    Posts
    12,461
    Blog Entries
    59


    Did you find this post helpful? Yes | No
    Quote Originally Posted by LoneWoLF View Post
    Funny you say that, cause the other side of the coin is the same way. The side counters clump all simple practitioners in the same class, but you dont say anything about them doing that.
    Once again, you use a broad brush. The 'side counters,' as a group, do no such thing. There are a vast numbers of methods that use side counts. Grouping them all together and assigning some attribute to them makes no sense. Personally, I dislike using side counts. But, there are situations where they possess enormous utility. For example, certain side bets. When you decide to make such labels, you close off possible opportunities.

    The point of this forum is to talk about modern AP methodologies. If you want the site to be limited to how to use HiLo; you're in the wrong site. As conditions worsen, we need to look at new opportunities. Obviously, some will not be useful. But, we don't stop opinions. We stop personal attacks, religion, and politics.
    "I don't think outside the box; I think of what I can do with the box." - Henri Matisse

  12. #12


    Did you find this post helpful? Yes | No
    Quote Originally Posted by Norm View Post
    Once again, you use a broad brush. The 'side counters,' as a group, do no such thing. There are a vast numbers of methods that use side counts. Grouping them all together and assigning some attribute to them makes no sense. Personally, I dislike using side counts. But, there are situations where they possess enormous utility. For example, certain side bets. When you decide to make such labels, you close off possible opportunities.

    The point of this forum is to talk about modern AP methodologies. If you want the site to be limited to how to use HiLo; you're in the wrong site. As conditions worsen, we need to look at new opportunities. Obviously, some will not be useful. But, we don't stop opinions. We stop personal attacks, religion, and politics.
    You just changed the whole subject matter though. I'm talking about side bets 'playing blackjack' in a shoe game. Side bets are not blackjack.

  13. #13
    Senior Member
    Join Date
    Dec 2011
    Location
    3rd rock from Sol, Milky Way Galaxy
    Posts
    14,158


    Did you find this post helpful? Yes | No
    Quote Originally Posted by LoneWoLF View Post
    You just changed the whole subject matter though. I'm talking about side bets 'playing blackjack' in a shoe game. Side bets are not blackjack.
    If they are on the blackjack table and you want to exploit both BJ and the side bet at an advantage then your count better be up to the task and it is blackjack.

Page 1 of 3 123 LastLast

Similar Threads

  1. Random Statistical Question
    By BramSToker in forum General Blackjack Forum
    Replies: 7
    Last Post: 01-27-2015, 02:19 PM
  2. Is there any significance to the 7's in REKO or strategy?
    By Blitzkrieg in forum General Blackjack Forum
    Replies: 6
    Last Post: 02-24-2014, 03:57 AM
  3. Replies: 5
    Last Post: 09-15-2007, 09:38 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

About Blackjack: The Forum

BJTF is an advantage player site based on the principles of comity. That is, civil and considerate behavior for the mutual benefit of all involved. The goal of advantage play is the legal extraction of funds from gaming establishments by gaining a mathematic advantage and developing the skills required to use that advantage. To maximize our success, it is important to understand that we are all on the same side. Personal conflicts simply get in the way of our goals.