[bulk] Resized Images (downscaled, upscaled)

@DanbooruBot: The bulk update request #1608 (forum #146249)...

2018-06-22T02:12:51-04:00

The bulk update request #1608 (forum #146249) has been approved by @Hillside_Moose.

@DanbooruBot: This bulk update request is pending automatic...

2018-06-22T00:03:44-04:00

This bulk update request is pending automatic rejection in 5 days.

@nonamethanks: create implication upscaled -> resized create...

2018-04-27T15:30:10-04:00

create implication upscaled -> resized
create implication downscaled -> resized

Link to request

Resubmitting after automatic rejection.

EDIT: This bulk update request is pending automatic rejection in 5 days.

EDIT: The bulk update request #1608 (forum #146249) has been approved by @Hillside_Moose.

@DanbooruBot: The bulk update request #1025 (forum #124918)...

2018-04-27T14:44:25-04:00

The bulk update request #1025 (forum #124918) has been rejected by @DanbooruBot.

@DanbooruBot: This bulk update request has been rejected...

2018-04-27T14:44:24-04:00

This bulk update request has been rejected because it was not approved within 60 days.

@BrokenEagle98: (╯‵□′)╯︵┻━┻FFFFFUUUUUUU-..... Forgot to...

2017-01-03T19:59:13-05:00

(╯‵□′)╯︵┻━┻

FFFFFUUUUUUU-.....

Forgot to account for animated... I went back and removed all md5 mismatch, downscaled and resized from all of those posts. If I missed any, please feel free to remove those tags... le sigh...

@evazion: It did get the biggest video, unless there's a...

2017-01-03T19:13:13-05:00

It did get the biggest video, unless there's a bigger one somewhere that I'm missing. Maybe @BrokenEagle98's script got confused.

@user_460797: I've used the bookmarklet when I uploaded it in...

2017-01-03T18:56:53-05:00

I've used the bookmarklet when I uploaded it in the first place. So it should've grabbed the biggest size available when uploading.

@evazion: Is it really downscaled? There's a thumbnail...

2017-01-03T18:54:09-05:00

Is it really downscaled? There's a thumbnail that is 600x450 but the largest video is 480x360, same as post #2415080.

600x450 .jpg: http://pbs.twimg.com/ext_tw_video_thumb/750308790306639872/pu/img/EsEXsC-0CpQOZ2a5.jpg
480x360 .mp4: https://video.twimg.com/ext_tw_video/750308790306639872/pu/vid/480x360/_ba6QNq_99D63GHV.mp4
240x180 .mp4: https://video.twimg.com/ext_tw_video/750308790306639872/pu/vid/240x180/k3cAYii7O76hu0qu.mp4
??? .m3u8: https://video.twimg.com/ext_tw_video/750308790306639872/pu/pl/rJxNGHjnuz5jR-xe.m3u8
??? .mpd: https://video.twimg.com/ext_tw_video/750308790306639872/pu/pl/rJxNGHjnuz5jR-xe.mpd

Twitter API response

~/src/danbooru $ rails runner 'puts TwitterService.new.client.status(750310569857519623).attrs.to_json' | jq .

{
  "created_at": "Tue Jul 05 12:49:14 +0000 2016",
  "id": 750310569857519600,
  "id_str": "750310569857519623",
  "text": "今日のお絵描き、艦これの朝潮改二。描いたついでにちょっと動かしてみた。 https://t.co/NjtZpKcOLR",
  "truncated": false,
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [],
    "media": [
      {
        "id": 750308790306639900,
        "id_str": "750308790306639872",
        "indices": [
          36,
          59
        ],
        "media_url": "http://pbs.twimg.com/ext_tw_video_thumb/750308790306639872/pu/img/EsEXsC-0CpQOZ2a5.jpg",
        "media_url_https": "https://pbs.twimg.com/ext_tw_video_thumb/750308790306639872/pu/img/EsEXsC-0CpQOZ2a5.jpg",
        "url": "https://t.co/NjtZpKcOLR",
        "display_url": "pic.twitter.com/NjtZpKcOLR",
        "expanded_url": "https://twitter.com/zzaigo/status/750310569857519623/video/1",
        "type": "photo",
        "sizes": {
          "large": {
            "w": 800,
            "h": 600,
            "resize": "fit"
          },
          "small": {
            "w": 340,
            "h": 255,
            "resize": "fit"
          },
          "medium": {
            "w": 600,
            "h": 450,
            "resize": "fit"
          },
          "thumb": {
            "w": 150,
            "h": 150,
            "resize": "crop"
          }
        }
      }
    ]
  },
  "extended_entities": {
    "media": [
      {
        "id": 750308790306639900,
        "id_str": "750308790306639872",
        "indices": [
          36,
          59
        ],
        "media_url": "http://pbs.twimg.com/ext_tw_video_thumb/750308790306639872/pu/img/EsEXsC-0CpQOZ2a5.jpg",
        "media_url_https": "https://pbs.twimg.com/ext_tw_video_thumb/750308790306639872/pu/img/EsEXsC-0CpQOZ2a5.jpg",
        "url": "https://t.co/NjtZpKcOLR",
        "display_url": "pic.twitter.com/NjtZpKcOLR",
        "expanded_url": "https://twitter.com/zzaigo/status/750310569857519623/video/1",
        "type": "video",
        "sizes": {
          "large": {
            "w": 800,
            "h": 600,
            "resize": "fit"
          },
          "small": {
            "w": 340,
            "h": 255,
            "resize": "fit"
          },
          "medium": {
            "w": 600,
            "h": 450,
            "resize": "fit"
          },
          "thumb": {
            "w": 150,
            "h": 150,
            "resize": "crop"
          }
        },
        "video_info": {
          "aspect_ratio": [
            4,
            3
          ],
          "duration_millis": 18000,
          "variants": [
            {
              "bitrate": 832000,
              "content_type": "video/mp4",
              "url": "https://video.twimg.com/ext_tw_video/750308790306639872/pu/vid/480x360/_ba6QNq_99D63GHV.mp4"
            },
            {
              "bitrate": 320000,
              "content_type": "video/mp4",
              "url": "https://video.twimg.com/ext_tw_video/750308790306639872/pu/vid/240x180/k3cAYii7O76hu0qu.mp4"
            },
            {
              "content_type": "application/x-mpegURL",
              "url": "https://video.twimg.com/ext_tw_video/750308790306639872/pu/pl/rJxNGHjnuz5jR-xe.m3u8"
            },
            {
              "content_type": "application/dash+xml",
              "url": "https://video.twimg.com/ext_tw_video/750308790306639872/pu/pl/rJxNGHjnuz5jR-xe.mpd"
            }
          ]
        },
        "additional_media_info": {
          "monetizable": false
        }
      }
    ]
  },
  "source": "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
  "in_reply_to_status_id": null,
  "in_reply_to_status_id_str": null,
  "in_reply_to_user_id": null,
  "in_reply_to_user_id_str": null,
  "in_reply_to_screen_name": null,
  "user": {
    "id": 549294672,
    "id_str": "549294672",
    "name": "イゴオ",
    "screen_name": "zzaigo",
    "location": "",
    "description": "適当にソシャゲなどに手を出しているぬるオタ。たまにしょぼい落書きを放り投げる模様。プレイ中のゲーム→艦これ/花騎士/政マニ",
    "url": null,
    "entities": {
      "description": {
        "urls": []
      }
    },
    "protected": false,
    "followers_count": 2209,
    "friends_count": 35,
    "listed_count": 56,
    "created_at": "Mon Apr 09 13:03:16 +0000 2012",
    "favourites_count": 15,
    "utc_offset": 28800,
    "time_zone": "Irkutsk",
    "geo_enabled": false,
    "verified": false,
    "statuses_count": 99,
    "lang": "ja",
    "contributors_enabled": false,
    "is_translator": false,
    "is_translation_enabled": false,
    "profile_background_color": "000000",
    "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
    "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
    "profile_background_tile": false,
    "profile_image_url": "http://pbs.twimg.com/profile_images/754399202893979648/33dutB05_normal.jpg",
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/754399202893979648/33dutB05_normal.jpg",
    "profile_link_color": "700000",
    "profile_sidebar_border_color": "000000",
    "profile_sidebar_fill_color": "000000",
    "profile_text_color": "000000",
    "profile_use_background_image": false,
    "has_extended_profile": false,
    "default_profile": false,
    "default_profile_image": false,
    "following": null,
    "follow_request_sent": null,
    "notifications": null,
    "translator_type": "none"
  },
  "geo": null,
  "coordinates": null,
  "place": null,
  "contributors": null,
  "is_quote_status": false,
  "retweet_count": 3762,
  "favorite_count": 6180,
  "favorited": false,
  "retweeted": false,
  "possibly_sensitive": false,
  "possibly_sensitive_appealable": false,
  "lang": "ja"
}

@Mikaeri: Looks like it might be a bug?

2017-01-03T17:32:13-05:00

Looks like it might be a bug?

@user_460797: Recent upload from me by using the bookmarklet...

2017-01-03T17:28:28-05:00

Recent upload from me by using the bookmarklet on
https://twitter.com/zzaigo/status/750310569857519623
http://danbooru.me/uploads/1278221

because post #2415080 is apparantly downsized.
So I thought it should work that I re-upload that video, but apparantly it doesn't.

@BrokenEagle98: Okay... what I could probably do then is just...

2017-01-02T23:26:18-05:00

Okay... what I could probably do then is just dump all of the "upscaled" images (larger on Danbooru, smaller at source) in a forum post somewhere, and people can go through them manually if they want.

I'll continue to tag (as I have tagged) "downscaled" images (smaller on Danbooru, larger at source), so that people can easily find those posts and check the source to determine if it's worth uploading and replacing the version on Danbooru.

Just as a status update, I finished going back and evaluating all of the past MD5_mismatch posts from Twitter, Tumblr and Nicoseiga, and tagged them as above when appropriate with "downscaled" and "resized".

Currently, I'm going through the Tinami posts and converting the image URL links to post links manually as there is no way to check if an image is full size or sample size without checking the post... :(

@Mikaeri: Ooh, informative! Thanks for the lengthy write...

2017-01-02T23:15:06-05:00

Ooh, informative! Thanks for the lengthy write up.

It's still a big judgement call in the end, though, unless you can recognize some obvious tell-tale artifact. There's a huge rabbit hole to jump down here if you want to, but please be aware that you have to understand digital image stuff pretty well to make a good call, and even then some cases are hard to tell. Not to be obnoxious, but I just don't want someone to read this post and then start flagging all kinds of stuff saying they upscaled/downscaled it and it kinda looked different.

So this makes for the convincing argument that md5 mismatches with larger file sizes may not actually be upscaled but instead originals that might've been just changed in the future by the site they were hosted on (twitter, tumblr from what we have), if I'm understanding this correctly. Quite honestly, the "originals" we have are very good as you say, although I'm not very well-versed in digital image stuff. I suppose I'll be of the opinion that we shouldn't tag images with higher resolutions from the source upscaled, and @BrokenEagle98 should probably be wary of it too.

As for the scaling algorithms, I definitely can see the difference, thanks for giving some examples!

waifu2x can try to remove jpeg artifacts with or without scaling.

Mhm, I checked it out after evazion made mention of that. Seems to be used a lot for making wallpapers or just larger image samples from original images that were deemed too small to use, but it can be used for that too.

(Haha, who am I kidding, nobody's going to read this whole post.)

Nah, I read the whole thing word for word. You're very well spoken, I'm glad I made a mention to you :)

EDIT: Just to add:

So archiving a third-party resize is generally not only wasteful but actually counter-productive - you could end up later in a situation where you could have produced an image using the original that is better than the previously scaled image. The original from the artist is the only non-reconstructable information, so it should absolutely be treated specially. You also can't trust third parties to resize with as high quality as they could. It's extremely common, for example, to downscale in a non-linear colorspace, which can destroy details such as the stars in a night sky. (See second example below.) Unfortunately, you can't always trust the artist in that respect either, but whatever you get from the artist is the best you're going to get.

This is my mindset, which is why I added such a line to the wiki page for waifu2x, albeit it's the best resize tool that exists right now for anime-style artwork. At the very least though, we can usually trust the artist's judgment (although it's a rarity here and there if an artist chooses to reupload a larger 'original' resolution than the one he uploaded previously on pixiv, for example).

With that said though, you are right on that it is still hard to tell. Gollgagh made a good point on post #2558234 that the original image uploaded by the artist (and we can't totally be sure if the artist revised it at some point) looks blatantly pixelated with scaling artifacts, which gives a somewhat convincing argument that what was uploaded previously might have been the original (although we'll never know given original uploader didn't link directly to the image URL)

@☆♪: Hmm, I actually got the mention message even...

2017-01-02T22:57:41-05:00

Hmm, I actually got the mention message even though the post doesn't link to me here. (And in the dmail, it is linked.)

If I had to guess, I'd say both posts came from a different source than they claim, rather than being upscales.

post #2436032 (tumblr): First point of interest, the listed source, although named .png, is actually a JPEG. The danbooru upload is a real PNG. But I don't think it's an upscale. The tumblr version has some artifacts that don't appear in the larger image. In fact, looking at the edges, the tumblr version resembles a downscale in a non-linear colorspace of the larger one. It could also be that the larger one was unsharp-masked, but that wouldn't have brightened the thin lines and would have amplified the jpeg artifacts as well. At the very least, if it is an upscale, it isn't an upscale of the tumblr image.

post #2517706 (twitter): Harder to tell, but I think again it's not upscaled. I can almost see it being a sharpened upscale, but I don't think the detail on the upper-left-corner knight's helmet would have be reproduced by that, and there's some jpeg-induced color distortion in the twitter one that would have infected an upscale and isn't present as much in the larger image.

We won't know what happened for sure unless we ask the uploaders. I'm not sure if this topic is the right place for that or not.

Mikaeri said:

Mmmm... quite honestly, I'm not sure if one can tell if such posts are third party resizes or artist resizes unless it's blatantly described in the source. Is the distinction that necessary though? It seems fairly unnecessary to me right now, although my mind may change if you can bring up a better reasoning.

Warning, wall of text incoming here.

Why the distinction is necessary: A different size produced by the artist is likely to be exported from the original (possibly vector) source at the new resolution rather than scaled. That's higher fidelity by definition than a raster scale and almost always subjectively better looking as well. Furthermore, research is always ongoing and technology always improving, but algorithms are unlikely to be forgotten, so if you do a resize now, you might be able to do it better in the future, but if you keep the original, you can always resize it at least as well as you can now whenever you need to. Also, if you need a different size later, it's better to resize from the original directly to the new size, rather than from an already-resized image, because the latter introduces rounding error every time you rasterize, as well as amplification of artifacts left by the scaling algorithm. (See first example below.) So archiving a third-party resize is generally not only wasteful but actually counter-productive - you could end up later in a situation where you could have produced an image using the original that is better than the previously scaled image. The original from the artist is the only non-reconstructable information, so it should absolutely be treated specially. You also can't trust third parties to resize with as high quality as they could. It's extremely common, for example, to downscale in a non-linear colorspace, which can destroy details such as the stars in a night sky. (See second example below.) Unfortunately, you can't always trust the artist in that respect either, but whatever you get from the artist is the best you're going to get.

How to tell: It's not always easy, which is one reason it's always nice to have the proper source of the image. But you can look for the artifacts caused by various resizing algorithms, or the amplification of jpeg artifacts if the original was a jpeg (or other lossy compression). In the case where you have a smaller image that you know is original, you can try upscaling that the best you can to the size of the maybe-original larger image, and see if it looks as good. If the larger image has actual detail that isn't present in the enlarged smaller one, it's more likely to at least not be an upscale of that image. You can also downscale the larger image to the size of the smaller one, and see if it looks better, which could happen if the smaller one was a starved jpeg, for example, but wouldn't happen if it had been created from the small one to begin with. It's still a big judgement call in the end, though, unless you can recognize some obvious tell-tale artifact. There's a huge rabbit hole to jump down here if you want to, but please be aware that you have to understand digital image stuff pretty well to make a good call, and even then some cases are hard to tell. Not to be obnoxious, but I just don't want someone to read this post and then start flagging all kinds of stuff saying they upscaled/downscaled it and it kinda looked different. (Haha, who am I kidding, nobody's going to read this whole post.)

Some examples of distortion caused by resizing:
post #2354838: resized directly to 800px wide and resized in 5px steps. What artifacts you would see in the second case depend on the algorithm used to scale, but at least I used the same algorithm in both cases here (EWA LanczosSharp). To give another example with different artifacts, here's the same with a Gaussian filter, which is blurry but doesn't have other artifacts like the ringing that Lanczos produces: direct and in steps. Quite a difference. Note that in either case, the aspect ratio is also distorted by resizing multiple times because every step has to round the size to full pixels. Of course scaling an image 60 times is going to end up a lot worse than 2 or 3, but I'm just trying to demonstrate that something is lost each time.
post #1891289: resized in linear colorspace and resized in sRGB (the default for the vast majority of software). Small bright points come out much dimmer than they're supposed to be. In images without many small bright points, the effect is less noticeable but still results in a darkening and slight blurring of edges, for example. The visual attractiveness of that effect can be debated, but it is unquestionably a less accurate reproduction of the original. Make sure you view the example images with 100% zoom (and look at the original at 100% as well), because even your browser most likely does it wrong, as does Danbooru's 850px sample. You might be amazed as you click zoom or view original and new stars appear.

Wow, I spent a lot longer on this than I intended to.

In addition to that though, the waifu2x tag is quite peculiar... Does it always do resizes? I think I've seen some posts tagged waifu2x where all that happens is the jpg gets converted to png like with post #2573016 and post #2573019

waifu2x can try to remove jpeg artifacts with or without scaling.

@evazion: > Mikaeri said: > > In addition to that...

2017-01-02T21:30:38-05:00

Mikaeri said:

In addition to that though, the waifu2x tag is quite peculiar... Does it always do resizes? I think I've seen some posts tagged waifu2x where all that happens is the jpg gets converted to png like with post #2573016 and post #2573019

I don't know much about it, but apparently it isn't just for upscaling, it can also smooth jpeg artifacts. See http://waifu2x.udp.jp/ and https://github.com/nagadomi/waifu2x. In those posts it looks like someone used it to remove the jpeg artifacts from the twitter samples.

BrokenEagle98 said:

Tumblr example: post #2436032

The source wasn't changed since it was originally uploaded. The strange thing is that the size on Danbooru is (1425x1425), whereas the maximum dimensions on Tumblr are 1280 in either direction, so... ¯\_(ツ)_/¯

Twitter example: post #2517706

I don't know what to make of that, other than the uploader uploading a file they got from one place but setting the source to Twitter/Tumblr. I wouldn't think that Twitter post is an upscale, it doesn't have that many artifacts for something that's double the resolution of an already artifact-y Twitter sample.

@Mikaeri: I wish I could say... maybe @evazion or @☆♪ can...

2017-01-02T19:32:02-05:00

I wish I could say... maybe @evazion or @☆♪ can offer their opinion on the matter?

EDIT: Oh, unicode, how wonderful you are…

@BrokenEagle98: So what the best method here... should those...

2017-01-02T19:13:49-05:00

So what the best method here... should those just be automatically tagged upscaled, or should they be posted onto this thread so that they can be looked over and manually tagged...?

@Mikaeri: Yup, it's just a question of if it's always...

2017-01-02T17:51:00-05:00

Yup, it's just a question of if it's always been like that. Has Tumblr ever allowed for larger uploads in the past? Because I can't remember a time there was.

As for Twitter, I've made up my mind and I'm pretty confident it's an upscale. Again, Twitter doesn't allow you to replace images in place like pixiv does, so either upcale or wrong source.

@tapnek: The maximum resolution is 1280px wide and...

2017-01-02T16:37:18-05:00

The maximum resolution is 1280px wide and 1920px tall.

@user_460797: Well, isn't the mximum resolution on Tumblr not...

2017-01-02T05:44:56-05:00

Well, isn't the mximum resolution on Tumblr not 1280*1280, although the image itself could be much bigger?