I’ve been trying to access a website and it’s basically impossible for me unless I use data from my cell phone provider. I’ve tried various things such as using a VPN, using Tor browser, and restarting my router and yet nothing has worked. I get this exact same error on my entire WiFi network. I decided to wait 24 hours and then ask on here just to make sure. This is frustrating because I’ve never gotten this error in the past. Any help or guidance would be appreciated. Thank you!
My guess would be they may have identified bot activity and fingerprinted you to stop it. Not sure how exactly but that seems the most obvious.
The short answer is you don’t. It’s THEIR problem. Their server is getting slammed and/or they’ve already hit their quota.
Are you accessing a web API or perhaps scraping a website?
If you’re working with a web API, then the docs should contain information about their rate limiting policy. This will explain how often within a certain time span you can access the API and what to do if you hit the limit.
If you’re scraping a website (or the API has no public rate limiting policy), then you simply need to add a delay between your requests.
How big that delay should be depends entirely on the website, so you’ll have to experiment. A couple hundred milliseconds is usually more than enough in my experience.
I guess that makes sense. I just hope I can access it again. Thanks for your help!
Then why is it working fine when I try to browse the site using my data plan?
Not at all. At the time, I was basically running an Auto Refresh extension that refreshed every 5 minutes, or 288 times a day. On top of that, I had something running that would load the website an additional 20 times, so about 308 times total a day. I don’t even think they have an API to work with. Any idea what could be the issue?
You’re just lucky they weren’t being hit hard at the time.
I would literally do that right after I initially attempted to browse the site, so that doesn’t make much sense.