With the record summer heat upon us all, it may be difficult to recall the Winter Olympics last February. But some of you might remember ice skater Ilia Malinin (aka “The Quad God”). He surprised almost everyone by failing to earn a medal in the Men’s Individual figure skating event. He had been considered almost a lock to win Gold. Although disappointed, in a post-event interview he said “you learn a lot more from failure than you do winning.” While few of us will ever reach his elite performance level, those words still apply to everyone.
The best lessons in life rarely come wrapped in success. Victories look great on a résumé, but they teach you surprisingly little. Mistakes, on the other hand, that’s where the value is. If you’ve only made one or two errors in your career, you probably weren’t trying hard enough.
If you stay inside the box, the world is a safe space. But step outside and, as Canadians say, you better get “elbows up.” You’ll get a few scrapes, but you’ll also collect wisdom that can’t be found in a classroom or training manual. Those scars? They’re your career compass.
When I hung up my shingle as a “casino consultant” (that’s a polite term for retired), I used this tagline: “I want to share some of my experiences and help you to prevent making all the mistakes I did over the years.”
In conversations with industry legends (like John Acres, Denny Garcia, Nick Greenwood, Phil Hannifin, Scott Olive, Charlie Lombardo, Bruce Rowe, Tyler Sciotto, Jean Venneman, and others), they occasionally discussed their “greatest hits.” But often, they painfully recalled (and sometimes chuckled) about what didn’t work: the promotions, products, designs, strategies or technologies that backfired. The innovations that failed. The close calls that could have been. The small disasters that happened.
Please forgive my blatant name dropping above, but I knew most of these folks when they were early in their careers. They often shared some of their less stellar moments (aka educational experiences) that helped them on the path to becoming innovators and pioneers. Those small missteps are just one reason why you probably know most of their names today.
In that spirit, below are just five of the dozens and dozens of my personal “less-than-proud” experiences. Each one came with a tuition bill from the “School of Hard Knocks,” but each earned one more credit toward a degree in successful operations.
In that spirit, below are just five of the dozens and dozens of my personal “less-than-proud” experiences. Each one came with a tuition bill from the “School of Hard Knocks,” but each earned one more credit toward a degree in successful operations.
1. SCOTCH TASTING FESTIVAL (Lesson: Unintended Consequences)
As a young marketing director at Fitzgeralds in Reno, NV, I was tasked with coming up with new and innovative promotions with an Irish or Celtic theme. In those days, Irish Whiskey wasn’t as trendy as today, but Scotch was. So, I organized a one-day Scotch Tasting Event with the local liquor distributor. For $20 (complimentary for our VIPs), patrons would receive three Red tickets, three Green tickets and one Gold ticket. For the Reds, you could receive a ¼ shot of a common blended scotch such as Johnny Walker, Dewars or Chevas. The Green tickets would get a premium blended (like a Johnny Walker Blue) or a decent single malt (think Glenfiddich, Talisker or Lagavulin). Your Gold ticket? That was good for a rarified gem like a single malt 30-year Macallan, a 40-year Dalmore or a 30-year Balvenie Rare Marriages.
The Scotch Tasting event was a big one-day success. Gaming revenues got a nice bump, and management was pleased. But then, we found that our casino VIPs (who in the past were very happy with a J&B and soda) were now demanding multiple neat shots of an aged Macallan at $4,500+ per bottle. Our comp costs soared. If only I had heeded the ancient credo that most hosts have learned: “Overly demanding VIPs are made, not born.”
2. VIDEO POKER PROS (Lesson: KYC, Know Your Customer)
I had never encountered many video poker “pros” before I started at the Atlantis Casino Resort in Reno. The property had dozens and dozens of loose video poker machines. It was one of the pros favorite hang outs. In those days, my stereotypical image of a poker “professional” was like the movie version of the low-life hustlers that lurked in the shadows of smoke-filled pool halls. I never understood how they even scraped together enough money to play nickel machines!
That said, they were dependable repeat visitors. They were just too skilled. They collectively could lower hold percentages dramatically and ruin good promotions. Most average video poker players give up two to five points of payback since they seldom play perfectly and make multiple small mistakes.
The pros gave up nothing. Unfortunately, because of this, I couldn’t offer the very loosest poker machines to our loyal average guests or our VIPs. The pros would jump on those games. It didn’t seem fair.
I came up with what I thought would be an innovative solution. I would offer extremely loose video poker machines only on high denomination games ($10, $25 and $100). Surely, I reasoned that none of these so-called pros could afford those higher stakes. It seemed obvious that only our loyal and beloved VIP High Rollers would play and enjoy them. Big mistake!
The machines got hammered immediately by those very pros I wanted to exclude. Unbeknownst to me at the time, almost all professionals had a lot more money than I knew, or big-money patron sponsors, or simply friends that would back their play if they came upon a great opportunity.
They found it with my new loose high-denom machines. After only two days, I had to shut them all down. We lost a fair amount of money, and I got an earful of guest complaints, not to mention some choice words from my boss.
Ever since, I’ve made it a point to befriend and get to know advantage players. They totally changed my outlook. They are very skilled, brilliant with math, and work hard at their craft. Most of them know our games far better than most casino operators, and they seldom, if ever, break the laws. Today, I have maintained several valuable friendships with these players, and they have helped me avoid many future mistakes that others made and are still making.
3. BATTERY BACKUPS (Lesson: Beware Single Points of Failure)
At Pechanga Resort Casino, where I capped my daily career, we prided ourselves on over-engineering everything. Locally, Southern California Edison (SCE) was a notoriously unreliable utility, with frequent voltage spikes and small outages.
Taking advantage of low natural gas prices, the casino installed an innovative natural gas turbine generator (which was developed in nearby San Diego). It was state-of-the-art and handled about a third of all our power needs. At the same time, we supplemented our two existing diesel generators with four additional units that could handle the entire property if needed with a week’s worth of fuel.
As a redundant measure, we also installed four giant Mitsubishi UPS units. These were massive battery-based Uninterruptible Power Supplies that were hooked to all the slots, computers and other critical components. These were to eliminate any transient surges, outages or fluctuating voltages from SCE and/or our generators. Lastly, and perhaps as overkill, we added hundreds of small UPS units to slot machines and the “network switches” in the machine bases to prevent any localized outages, broken cables or slot system interruptions. No one in the world had a more robust or efficient backup power solution.
But then there was Murphy’s Law. One day we noticed a small flicker of our lights. The facility engineers reported that there had been a small blip from SCE, but it was now over. We were back on good “shore power” again.
Worryingly, they said that some of their meters were still showing a power outage. They did note that our gas turbine was working efficiently. Just as a precaution, they fired up all six generators in case there was something unexpected. The four mega-UPS units were fully charged and working as intended. But within the next 30 minutes, all our slot machines started going off-line, along with the lights, air conditioning, computers, escalators and elevators. Despite our planning and millions of dollars spent, we were in the dark: both figuratively and literally.
An hour or so later, we found the problem. SCE was fine. So were our generators and the innovative gas turbine. The UPS units worked as intended, but they could only handle the full load for about 30 minutes. We unfortunately discovered that the power company, the turbine, the generators, and all the UPS units fed their inputs to a single distribution “switch” that then fed the entire property. When the first power bump hit, that distribution switch turned off since it was plugged into the wall. That wasn’t supposed to happen because it had an internal automotive-like 12-volt battery to keep it running if power ever failed. For years it had worked perfectly, so no one had ever checked to see if that battery was in good shape. It wasn’t. The switch wouldn’t restart and had gone completely dead. And so had we.
It was a lesson learned about single points of failure and keeping your batteries charged.
4. RON HARRIS (Lesson: Greed is good … at revealing crooks)
If you do a Google search for Ron Harris, you’ll learn that he was a computer technician for the Nevada Gaming Control Board in the early 1990s who ended up in handcuffs and eventually prison for cheating Keno machines in Atlantic City. What you won’t easily find is that he also cheated dozens of slot machines, including several at my property in Reno.
Nevada regulators annually audited slot machine EPROM[i] chips to verify their authenticity using state-issued laptops. In a shock to them, Harris had secretly programmed those laptops to write new software to the EPROM chips, instead of just reading them. The duped agents then re-installed the rigged chips back into our games unaware that they were doing so.
Later, Harris’ group of scammers would enter a series of programmed bets into the machine in a specific pattern (like 1 coin, 3 coins, 2 coins, 3 coins, 1 coin) and the game might hit a jackpot.
Here’s the brilliant part. All casinos do regular reports to verify that games are performing correctly and not over- or under-holding. To counter this, Harris’s new EPROM software cheated all regular slot players by making sure they never won anything. Whenever his partners entered their “cheat codes,” the game would not pay off until it had ripped off enough honest players to cover the false jackpots. Once that bankroll was established, the cheats could then make the machine pay. Therefore, our analytics would show the machine was performing perfectly.
Harris was only caught because he got greedy and couldn’t resist the potential $100,000 Keno jackpots on the New Jersey boardwalk. Nevada regulators were humiliated. They changed their audit procedures immediately, making sure that all future laptops could only read chips, not write to them. They placed Harris in their infamous Black Book for life.
He was not the first, nor will he be the last crook to be caught because of greed.
5. RUSSIAN HACKERS (Lesson #1: It Is Good To Share)
Fast forward to 2014, and one of the most widely reported slot machine scams was underway. You’ve surely read about the ring of Russians who cracked the random number generator (RNG) algorithms on older Aristocrat slot machines. They hit casinos across the U.S. And, of course, my property was a victim.
Here’s where sharing and professional relationships paid off. I first met Darrin Hoke in Reno, NV in the mid-1990s. He was a security/surveillance pro at Circus Circus Casino in Reno when I was down the street at the Eldorado Casino Resort. We also taught gaming classes together at the local university. Hoke was always an anomaly within his discipline. He strongly believed in sharing his knowledge and data among his peers, even if they were competitors. That was in sharp contrast to many siloed casino executives who seem to prefer remaining isolated with no desire to talk with any outsiders.
Back to the Russians: In early 2014, Lumiere Place Casino in St. Louis, MO noticed that some of their machines were losing money, even though no big jackpots had been hit. At the same time, Hoke was the director of surveillance at L’Auberge du Lac Casino Resort in Lake Charles, LA. From reports, he started investigating similar slot machine activity as reported in St. Louis and elsewhere by examining surveillance footage. He was eventually able to identify 25 operatives who seemed suspicious, but he and others had not yet determined exactly what they were doing. Doggedly, he tracked the group’s hotel registration records. Thus, he identified two Russian suspects who had stayed in Temecula, CA earlier that same June (where I was the VP of Slot Operations at Pechanga).
Based on his tips, our Surveillance team spotted them quickly in our archived digital recordings from the previous month. But we only had wide shots of their play. Our team spent hours reviewing the footage of these two. We could only notice that the candle lights on the games they were playing had some unusual blinking patterns, and that each of these players always had one hand in their pants pocket.
We had been studying this footage for weeks, when on July 14th our surveillance operatives spotted one of these Russians who’d returned to play. This time our cameras zoomed in and got close ups from multiple angles. We also called agents from the California Department of Justice to make an arrest.
We nabbed the player and held his hands behind his back so he couldn’t destroy any evidence. He had two mobile phones in his pockets. Using his key fob, we found his car in the parking lot with four more phones and stacks of cash in the trunk. It was one of the first arrests in this case, and it led to many more across the country. We collected evidence and forwarded it to Aristocrat, GLI and the FBI, who helped unravel this scam.
In brief summary, this team had found flaws in these older game’s RNGs that allowed them to hit the “Spin” button at the exact times forecast by their vibrating cell phones. This timing data dramatically improved their ability to hit multiple bonus jackpots which were below the then $1,200 reporting thresholds. We only caught them due to the work and sharing of intelligence by Hoke and others.
(Lesson #2: Don’t Overreact) – At Pechanga (unlike St. Louis), all the machines being cheated were still performing well above house averages. While most casinos across the country got rid of these games once they heard the arrest reports, we simply modified our analytical systems to immediately sound alerts on anomalies in the bonus rounds, not just the large jackpots. That meant that we could quickly stop and arrest anyone trying this scam in the future. Better yet, we were able to keep our very popular and profitable Aristocrat video machines on the floor when others had removed them. Don’t have a knee-jerk reaction to bad news
Postscript:
While they are not part of my personal “Hall of Shame,” here are another half dozen instructive “fails” my colleagues have shared:
- Filling the “Regular Unleaded” storage tank at the tribal gas station with “Diesel” that resulted in multiple casino guests’ cars being disabled. (Lesson: Install better warning labels on tanks and hoses.)
- Misplacing a decimal point that resulted in a $10.00 free play offer being issued as a $1,000 perk. (Lesson: Check and double check before hitting the promotional “Execute” button.)
- Making the “hopper empty” error code of “3300” appear in the same window as the “Win” meter on older spinning reel slot machines. It resulted in hundreds of angry players who demanded “their” $3,300 payout. (Lesson: Test your games with real-world players before releasing them.)
- Failing to notice that one of the Keno balls had fallen out of the rack when the blower ball hopper was loaded. The Keno game had a record hold that day, but it was more than offset by angry guests who noticed that one number never appeared. There was also a very large fine forthcoming from gaming regulators. (Double check everything three times!)
- Assuming that playing card manufacturers would never produce decks that were cut in such a way that even amateurs could cheat at Baccarat. (Lesson: Consistently keep informed on industry happenings and never assume that everything is alright.)
- Without mentioning any specific property by name, having to do a multi-million-dollar remodel of your new main entrance because you “discovered” that Asian guests didn’t like walking into the mouth of a lion. (Lesson: Don’t forget to check cultural sensitivity.)
Joseph Sohm / Shutterstock.com
Good luck in your future. They like to say in Silicon Valley, “Fail Forward, Fail Fast.” It’s a catch phrase to remind them to constantly move ahead while not wasting years perfecting an idea that was weak from the start. They even celebrate their misadventures with an annual event called FailCon. It is a concept that has now spread to similar conferences held around the world.
I like to modify their approach and urge you to consistently try new things in our gaming world, but also carefully monitor and analyze the results of those efforts and quickly ‘course correct’ if needed.
Be forewarned that all the advice above only works well if your boss or employer also appreciates the future value of your educational miscues. Hopefully they are as enlightened as Ikea, which actually rewards risk takers who occasionally stumble.
Mistakes sting, but they also teach in ways successes never will. Skater Ilia Malinin learned a lot from his Olympic misfortune. The following month, he won the World Championships held in Prague by an historically-wide margin with a near perfect routine that included an “unscored” back flip finale.
So, don’t be afraid to step outside the box occasionally. You’ll get some bruises, but that’s where the breakthroughs can also happen.
# # #
[i] – EPROM is an acronym for Erasable Programmable Read-Only Memory. In early slot machines, they were computer chips that controlled the basic functions of the game, including jackpot payouts. Internal auditors can check to make sure that they have not been altered by comparing their “electronic signature” to those provided by the original manufacturer. In the Harris case, he also manipulated these signature “checksums” on his phony firmware to falsely mimic the right numbers. Brilliant, but bad.
