Communities

Writing
Writing
Codidact Meta
Codidact Meta
The Great Outdoors
The Great Outdoors
Photography & Video
Photography & Video
Scientific Speculation
Scientific Speculation
Cooking
Cooking
Electrical Engineering
Electrical Engineering
Judaism
Judaism
Languages & Linguistics
Languages & Linguistics
Software Development
Software Development
Mathematics
Mathematics
Christianity
Christianity
Code Golf
Code Golf
Music
Music
Physics
Physics
Linux Systems
Linux Systems
Power Users
Power Users
Tabletop RPGs
Tabletop RPGs
Community Proposals
Community Proposals
tag:snake search within a tag
answers:0 unanswered questions
user:xxxx search by author id
score:0.5 posts with 0.5+ score
"snake oil" exact phrase
votes:4 posts with 4+ votes
created:<1w created < 1 week ago
post_type:xxxx type of post
Search help
Notifications
Mark all as read See all your notifications »
Q&A

Welcome to Codidact Meta!

Codidact Meta is the meta-discussion site for the Codidact community network and the Codidact software. Whether you have bug reports or feature requests, support questions or rule discussions that touch the whole network – this is the site for you.

Post History

81%
+7 −0
Q&A Is it okay to scrape Codidact for personal tools?

As a general rule: yes, for personal use there's no issue with that. Caveats: First & foremost, this is a general rule - if we discover requests or usage that looks like a problem we'll want...

posted 1y ago by ArtOfCode‭  ·  edited 1y ago by ArtOfCode‭

Answer
#2: Post edited by user avatar ArtOfCode‭ · 2023-08-09T19:01:54Z (over 1 year ago)
  • As a general rule: yes, for personal use there's no issue with that. Caveats:
  • * First & foremost, this is a general rule - if we discover requests or usage that looks like a problem we'll want to know what's going on;
  • * CloudFlare sits in front of all the public communities and will take issue with anything that looks spammy - we have only limited control over exactly what it blocks or not;
  • * Please be a considerate scraper: consider that our poor server has to render every bit of content you request[^1] and limit yourself to no more than a couple of requests per second at _absolute_ maximum;
  • * Our frontend HTML can (and does) change frequently and without notice - be aware scraping may break often and require maintenance.
  • However, I'd also encourage you (and anyone considering this kind of thing) to get involved - contribute back as much as you can in return, if you're able to:
  • * If there are ways in which you'd like to see things (posts, notifications) that aren't yet possible, create or support feature requests for them.
  • * If you can _scrape_ frontend code you probably have a decent idea of _writing_ it: consider sending a PR to improve the UI yourself.
  • * If you can write Ruby (and if you can write Python, Ruby's not hard to pick up), you might be able to help improve backend code or implement what you're looking for yourself.
  • [Codidact Collab](https://collab.codidact.org) is available if you'd like to contribute but don't know how to get started; if you're looking to contribute but aren't sure what to work on, there are some [easier issues to start with](https://github.com/codidact/qpixel/issues?q=is%3Aopen+label%3A%22meta%3A+good+first+issue%22+sort%3Aupdated-desc) and [issues we're looking for contributors for](https://github.com/codidact/qpixel/issues?q=is%3Aopen+label%3A%22meta%3A+help+wanted%22+sort%3Aupdated-desc) on the repo.
  • As a general rule: yes, for personal use there's no issue with that. Caveats:
  • * First & foremost, this is a general rule - if we discover requests or usage that looks like a problem we'll want to know what's going on;
  • * CloudFlare sits in front of all the public communities and will take issue with anything that looks spammy - we have only limited control over exactly what it blocks or not;
  • * Please be a considerate scraper: consider that our poor server has to render every bit of content you request[^1] and limit yourself to no more than a couple of requests per second at _absolute_ maximum;
  • * Our frontend HTML can (and does) change frequently and without notice - be aware scraping may break often and require maintenance.
  • However, I'd also encourage you (and anyone considering this kind of thing) to get involved - contribute back as much as you can in return, if you're able to:
  • * If there are ways in which you'd like to see things (posts, notifications) that aren't yet possible, create or support feature requests for them.
  • * If you can _scrape_ frontend code you probably have a decent idea of _writing_ it: consider sending a PR to improve the UI yourself.
  • * If you can write Ruby (and if you can write Python, Ruby's not hard to pick up), you might be able to help improve backend code or implement what you're looking for yourself.
  • [Codidact Collab](https://collab.codidact.org) is available if you'd like to contribute but don't know how to get started; if you're looking to contribute but aren't sure what to work on, there are some [easier issues to start with](https://github.com/codidact/qpixel/issues?q=is%3Aopen+label%3A%22meta%3A+good+first+issue%22+sort%3Aupdated-desc) and [issues we're looking for contributors for](https://github.com/codidact/qpixel/issues?q=is%3Aopen+label%3A%22meta%3A+help+wanted%22+sort%3Aupdated-desc) on the repo.
  • [^1]: Yes, yes, API better etc etc. It's on the list.
#1: Initial revision by user avatar ArtOfCode‭ · 2023-08-08T18:33:06Z (over 1 year ago)
As a general rule: yes, for personal use there's no issue with that. Caveats:

 * First & foremost, this is a general rule - if we discover requests or usage that looks like a problem we'll want to know what's going on;
 * CloudFlare sits in front of all the public communities and will take issue with anything that looks spammy - we have only limited control over exactly what it blocks or not;
 * Please be a considerate scraper: consider that our poor server has to render every bit of content you request[^1] and limit yourself to no more than a couple of requests per second at _absolute_ maximum;
 * Our frontend HTML can (and does) change frequently and without notice - be aware scraping may break often and require maintenance.

However, I'd also encourage you (and anyone considering this kind of thing) to get involved - contribute back as much as you can in return, if you're able to:

 * If there are ways in which you'd like to see things (posts, notifications) that aren't yet possible, create or support feature requests for them.
 * If you can _scrape_ frontend code you probably have a decent idea of _writing_ it: consider sending a PR to improve the UI yourself.
 * If you can write Ruby (and if you can write Python, Ruby's not hard to pick up), you might be able to help improve backend code or implement what you're looking for yourself.

[Codidact Collab](https://collab.codidact.org) is available if you'd like to contribute but don't know how to get started; if you're looking to contribute but aren't sure what to work on, there are some [easier issues to start with](https://github.com/codidact/qpixel/issues?q=is%3Aopen+label%3A%22meta%3A+good+first+issue%22+sort%3Aupdated-desc) and [issues we're looking for contributors for](https://github.com/codidact/qpixel/issues?q=is%3Aopen+label%3A%22meta%3A+help+wanted%22+sort%3Aupdated-desc) on the repo.