Feature Support for Major Source Sites

@Mikaeri: bumperino. Monappy topic: topic #14771. Before...

2017-12-06T18:23:14-05:00

bumperino. Monappy topic: topic #14771.

Before I get any lazier, I guess I should start writing the howto for bcy, as that is by far the fastest growing "new" site that users here might be interested in. It's become significantly easier to sign up for bcy (as you can just login/cross-site signup with a weibo account).

Will also rename to howto:bcy.net (since that's what's actually on their logo, and is less confusing than the pinyin romanization).

@Mikaeri: http://horne.red/ * RocketNews24 Article This...

2017-08-24T15:56:49-04:00

http://horne.red/

RocketNews24 Article

This site seems to be owned by same company as Nijie, and thus shares practically the same interface. Only difference is the type of content it has a primary focus on, which seems to be aimed at fujoshi.

@evazion Do you think we'd be able to add support for the site given our support for Nijie? I don't know if anyone has ever uploaded from it yet (I haven't checked), but some links do exist in a few artist entries.

@Mikaeri: Yeah, I remember now. The site was jimdo....

2017-07-23T14:06:05-04:00

Yeah, I remember now. The site was jimdo. Images hosted on a jimdo site are located in https://image.jimcdn.com/

Example:

Couldn't find any documentation otherwise of this.

What else to remark... right, so I do have some free time after I finish a upload batch I have planned, and some new candidates to add to forum #133845 have popped up.

In addition to stuff I have shelved to write:

There are these:

Mihuashi is the most recent high interest discovery. Most images from Mihuashi match these two searches:

source:https://images.mihuashi.com/ -- direct link
source:https://www.mihuashi.com/ -- HTML link (can redirect to artist homepage or activities feed)

If anyone wants a howto written for any of these sites sooner than later, I'd love to know so I'll be putting my efforts someplace that won't be wasted.

@Mikaeri: > CodeKyuubi said: > > I've actually never...

2017-07-18T04:26:12-04:00

CodeKyuubi said:

I've actually never seen /bmiddle/ before, it's always /mw690/ or /mw1024/, and on a single occasion I saw /woriginal/ which is /large/ except with horrifying compression artifacts, for whatever reason.

For naver, you should always be able to find it under the mainframe element, but only if you expand the image to its full size at least once, if I'm remembering correctly. Life would be so much easier if they didn't try to disable right-clicking.

Yarr, that's why I mention the "Click on the image to open up the full size image viewer if it's available." part. Sometimes you can, sometimes you can't. Depends on the blog, but if you can't then it's usually already loaded to begin with.

Were there other sites though? I recall there was one site like Cloudinary (cloud storage) but I can't remember off the top of my head what it was. URL structure is extremely similar.

@CodeKyuubi: > Mikaeri said:I've actually never seen...

2017-07-18T04:06:29-04:00

Mikaeri said:

I've actually never seen /bmiddle/ before, it's always /mw690/ or /mw1024/, and on a single occasion I saw /woriginal/ which is /large/ except with horrifying compression artifacts, for whatever reason.

For naver, you should always be able to find it under the mainframe element, but only if you expand the image to its full size at least once, if I'm remembering correctly. Life would be so much easier if they didn't try to disable right-clicking.

@Mikaeri: I've seen uploads from a few rarer sites so I'm...

2017-07-18T03:01:08-04:00

I've seen uploads from a few rarer sites so I'm curious if anyone may still want howto's on more obtuse yet active sites that don't see a whole lot of usage. Here are some candidates along with extra notes:

Wix -- I already explained how to get non-sample versions in the OP, but uploading from Wix sites seem largely incidental since there are so many different ways a website can link their work (blog, homepage, etc).
Weibo -- This is a pretty easy one, actually. Just replace /bmiddle/, /mw690/, or whatever part of that URL reads after sinaimg.cn with /large/. post #2790775 is a recent example.
Cloudinary -- Information here. tl;dr strip everything between /upload/ and the file's public id (which includes folders if there are any). Although you may want to leave in post versions or play with them if you think there's changes (/v1/, /v2/, etc.)
Naver -- Naver is a real bitch to get full sizes from. If you have Chrome DevTools, open up your console, go to Sources. Click on the image to open up the full size image viewer if it's available. Then look for the images under the domain blogfiles.naver.net. Most of the time this will also be hidden under the mainFrame element.

@BrokenEagle98: AFAIK, the API from E-hentai is pretty much...

2017-06-11T00:00:41-04:00

AFAIK, the API from E-hentai is pretty much useless as far as scanning for image samples or MD5 mismatches.

Ref: https://ehwiki.org/wiki/API

However, if there were a way to get the filesize and MD5 hash for the fullsize image and all of the sample images, that would be useful. As that's not available, I was reduced to having to use HTML grabs.

@Grahf: > Mikaeri said: > > That could be it. Although...

2017-06-07T16:26:09-04:00

Mikaeri said:

That could be it. Although perhaps this discussion would be better happening in private -- can @Grahf maybe get in touch with the admin about that? Because although this is for honest purposes, y'no how it goes -- people may snoop in over it.

I sent a PM to Tenboro asking if there's anything he'd be comfortable sharing. It might be a while before I receive a response though. I'll keep you guys posted.

EDIT: He replied back saying "Some botlike behavior has rather aggressive ban triggers, but it depends of course on what they are trying to do. The API, for example, is exempt but still rate-limited to some extent."

I think that it might be better if there were more directed questions that I could ask potentially. So if anyone wants to post or send some to me I can ask on everyone's behalf. How about I wait a day or two then send any questions I get off.

@Mikaeri: > BrokenEagle98 said: > > Just for reference,...

2017-06-06T23:36:39-04:00

BrokenEagle98 said:

Just for reference, the last time I tried my hand at Ehentai, it IP banned me for a day after only 6 or so GETs spaced at least 2 minutes apart. To call their system trigger-happy would be severely downplaying it... :/

However, with the above I was only doing a GET to the HTML page. Although I haven't tested it yet, I'm wondering if Ehentai requires a full browser simulation to avoid the ban detection, i.e. downloading everything from the page and making all of the appropriate HTTP calls that a browser would. To lend credence to the above point, I once did a test where I loaded at least 10 tabs in under 30 seconds all to different images in different galleries, and nothing was triggered.

That could be it. Although perhaps this discussion would be better happening in private -- can @Grahf maybe get in touch with the admin about that? Because although this is for honest purposes, y'no how it goes -- people may snoop in over it.

@BrokenEagle98: > Mikaeri said: > > I get the feeling E-hentai...

2017-06-06T18:19:38-04:00

Mikaeri said:

I get the feeling E-hentai support will be wonky though, given how strict they are with auto-downloaders and other things like that. If danbooru is limited to one account for access, it might trigger those same blocking schemes. Maybe I should put question marks there instead.

Just for reference, the last time I tried my hand at Ehentai, it IP banned me for a day after only 6 or so GETs spaced at least 2 minutes apart. To call their system trigger-happy would be severely downplaying it... :/

However, with the above I was only doing a GET to the HTML page. Although I haven't tested it yet, I'm wondering if Ehentai requires a full browser simulation to avoid the ban detection, i.e. downloading everything from the page and making all of the appropriate HTTP calls that a browser would. To lend credence to the above point, I once did a test where I loaded at least 10 tabs in under 30 seconds all to different images in different galleries, and nothing was triggered.

@Grahf: Well, there's a few things that I know that...

2017-06-06T16:41:50-04:00

Well, there's a few things that I know that would be relevant for EH.

I believe that all images are downscaled past a certain point unless you've made a minimum donation or grinded out the site currency for a particular "perk" as it were.

I know that when I uploaded post #2740441 for example that I could just go into the gallery, onto the image, then view the image and initiate the bookmarklet from there and it did upload properly, although I did have to change the source after the fact from the jargon that the actual link was to the gallery page itself.

At the same time, I might be able to get away with doing that because I have those aforementioned perks. Anyone just browsing the site without might not be so lucky.

Truth be told, most of the time whenever I want to upload something to danbooru that I find in the galleries, 99% of the time I download it and then just manually upload the images because there's less inherent risk in doing so.

That being said if there's anything in terms of questions or more detailed information I can try and help provide it. I might have to take any real technical minded stuff to the site admin though, and I'm unsure if they'd be willing to answer. They might, but they also might not.

@Mikaeri: Some notes about Nijie: * Bookmarklet works...

2017-06-06T16:32:46-04:00

Some notes about Nijie:

Bookmarklet works from view.php but not from view_popup.php
Unsure if the bookmarklet will fix the source link from view.php when opening image in a new tab, but it definitely doesn't from view_popup.php
Batch bookmarklet does not currently work with Nijie.

Notes on Medibang:

Sample size has /440_330/ in the URL -- there may be others. Sample link for post #2728375 looks like the following:

https://dthezntil550i.cloudfront.net/gr/latest/gr1704160605552690002262457/440_330/6e88ca68-5996-48f0-a6e8-efa75252173d.png

Original is:

https://dthezntil550i.cloudfront.net/gr/latest/gr1704160605552690002262457/6e88ca68-5996-48f0-a6e8-efa75252173d.png

EDIT: Nijie support has been added, so points are moot now.

@Mikaeri: I get the feeling E-hentai support will be...

2017-06-06T16:18:36-04:00

I get the feeling E-hentai support will be wonky though, given how strict they are with auto-downloaders and other things like that. If danbooru is limited to one account for access, it might trigger those same blocking schemes. Maybe I should put question marks there instead.

@Grahf might know more... I hope.

@Type-kun: I don't think E-hentai is unsupportable, we can...

2017-06-06T16:12:07-04:00

I don't think E-hentai is unsupportable, we can always parse the html if things come to it, but it requires a lot of investigation, like any non-pixiv-lookalike source. Ehentai samples are a common problem by the way, I've seen it occur multiple times, because people assume that viewer shows the original and don't bother to look in the bottom. See also issue #2413.
By the way, support for sadpanda will come along automatically if we manage to do this, but it requires an ehentai account which allows the access, which might also be a problem unless someone has a spare one to sacrifice or knows exactly how to get said access fast.

@Mikaeri: I think it's useful to have a topic where we...

2017-06-05T17:30:33-04:00

I think it's useful to have a topic where we can currently see and discuss the progress of feature support for many of the major sites that we upload from, so I've composed the table below:

Legend:
◯ - supported
✕ - unsupported
― - unsupportable (to the best of my knowledge), or unnecessary
? - not sure

-	Pixiv	Twitter	Nicoseiga	Nijie	Tinami	Tumblr	Pawoo	DeviantArt	ArtStation	Drawcrowd	bcy.net	Medibang	E-Hentai	Yandere Konachan
Works with link/bookmarklet (HTML, single)	◯	◯	◯	◯	✕	◯*	◯	◯	◯	✕	✕	✕	―	✕
Works with bookmarklet (batch)	◯	◯	―**	◯	―**	◯*	◯	―**	◯	✕	✕	―**	―	―**
Works with link/bookmarklet (direct image URL)	◯	◯	◯	◯	✕	◯*	◯	◯	◯	◯	◯	◯	―	◯
Corrects sample sizes	◯	◯	◯	―	✕	◯	◯	◯	◯	✕***	✕***	✕***	✕	◯
Supports direct image URL as source	◯	―	◯	✕	✕	―	―	◯	✕	✕	✕	✕	―	✕
Bookmarklet fixes source link (to HTML with proper referrer)	―	◯	―	◯	✕	◯	◯	―	◯	✕	✕	✕	―	◯
Translated tags	◯	◯	◯	◯	✕	◯	◯	◯	◯	✕	✕	✕	―	―
Commentary fetch	◯	◯	◯	◯	✕	◯	◯	◯	◯	✕	✕	✕	―	―
image sample/md5 mismatch/bad id scripts run regularly	◯	◯	◯	◯	◯	◯	◯	◯	◯	◯	✕	✕	✕	◯
Replacement scripts run regularly (topic #14063, replaceme)	◯	✕	✕	―	✕	✕	✕	✕****	✕	✕	✕	✕	✕	✕
Search metatag available (e.g. pixiv:62729688)	◯	―	✕	✕	✕	✕	✕	✕	✕	✕	✕	✕	―	✕
Howto available	◯	◯	◯	◯	◯	◯	◯	◯	◯	◯	✕	✕	✕	✕
Userscript support	◯	?	✕	✕	✕	?	?	?	?	?	✕	✕	?	?
API documented	◯	✕	✕	✕	✕	✕	◯	✕	✕	✕	✕	✕	―	?
Indexed by SauceNAO	◯	✕	◯	◯	✕	✕	◯	◯*****	✕	✕	◯	◯	◯	◯
Indexed by ascii2d	◯	◯*****	◯	◯	◯	✕	✕	✕	✕	✕	✕	✕	✕	✕

* Does not yet work with custom domains.
** These sites use a single entry for each image.
*** /large/ on Drawcrowd, /w650 on BanCiYuan, /440_330/ on Medibang
**** Not possible to replace from Amazon S3 yet.
***** Very partially indexed.

If anyone else has more ideas (additional sites and what not) or ways to add support then feel free to contribute. This doesn't cover all of the available sites -- I've left some out because they're either very rarely uploaded from (e.g. weibo, fc2) and/or defunct websites (e.g. Twitpic).

Wix support might be added. I'm in favor of it. There's been a number of artists that have been using wix to provide images of their artwork (post #2743138, post #2744357, etc), although it's still a rarity. Some notes:

To avoid uploading image samples from Wix, you need to strip out a lot of the link parameters. A link may look like something like:

https://static.wixstatic.com/media/cffccb_b3ff68b920634ed79575b21033141a97~mv2.jpg/v1/fill/w_391,h_1072,al_c,q_85,usm_0.66_1.00_0.01/cffccb_b3ff68b920634ed79575b21033141a97~mv2.webp

Whereas the correct link to the image (for the full-size) is:

https://static.wixstatic.com/media/cffccb_b3ff68b920634ed79575b21033141a97~mv2.jpg

Source link should point to the website afterwards if necessary, or can be left directly if from the artist's homepage, supposedly.

Edits

06.06.2017 - Fixed a few mistakes, added new rows (Fixes sample sizes, Userscripts, API).
06.10.2017 - Pawoo API link added.
06.21.2017 - Extra support for multiple sites added. Translated tags row added.
06.22.2017 - Replacement scripts run regularly row added.
06.25.2017 - Some stuff updated. Buncha support added for Nijie, Pawoo, among other sites.
07.13.2017 - Better support for Tumblr (bookmarklet/upload page, commmentary, tags).
07.18.2017 - howto:nijie written, got that squared off.
08.04.2017 - Support for Tumblr does not include custom domains.
09.12.2017 - Added new row: Indexed by ascii2d
12.06.2017 - long time no update. Changed name for bcy.net for now.