external pages or resources returned a 403 HTTP status code
(1) By anonymous on 2021-01-24 14:02:30 [link]
Hi, I have a number of pages on my website that are linking to your homepage, but when I run my website service to manage the website SEO, it is returning an "external pages or resources returned a 403 HTTP status code" error message. Pointing people to your site as using SQLite database to demonstrate how easy it is to use and do updates etc. Can you advise if you have blocked the service providers bot from retrieving your webpage? I am using SEMRUSH by the way. Thanks, Joe
(2) By Kees Nuyt (knu) on 2021-01-24 22:42:14 in reply to 1 [link]
Yes, sqlite.org has protections in place to prevent bots from (recursively) iterating over all links in the web site. In my opinion, your tool should not try to follow external links. Any content of sqlite.org is **not** part of **your** website and should not be analyzed by your tool. It is perfectly fine to have hyperlinks to sqlite,org on your site, and your users are free to click them. But the tools should not do that. Just my two cents.
(3) By David Jones (vman59) on 2021-01-25 12:16:25 in reply to 2 [link]
I download the documentation bundle (e.g. sqlite3-doc-3340100.zip) and serve the files it contains with my own web server for local consumption. There only seem to be a few omissions from what's on the sqlite.org site and few of the links are hard coded to sqlite.org or external links.
(4) By Kees Nuyt (knu) on 2021-01-25 15:35:42 in reply to 3 [link]
I have a similar construction, but I generate the site from a local clone of the sqlitedoc repository. Then I run a postprocessing script over the generated html pages to replace any references to sqlite.org with links to my local site.
(5) By David Jones (vman59) on 2021-01-25 16:48:46 in reply to 4
My web server can directly serve a zip file as a document branch so I do minimal processing of it. The direct references to sqlite.org seem to be to files not included in the doc bundle so there are no local files to rewrite them to.