Resource Raw Text Remover

LysUltima · Jan 14, 2018

Here.
If you translate on one document under/over raw lines, this should be helpful.
Unless the raws have a line that's more English than Japanese/Chinese/Korean, this should work 100% of the time.

Haxagen · Jan 14, 2018

LysUltima said: ↑

Here.
If you translate one one doc under/over raw lines, this should be helpful.
Unless the raws have a line that's more English than Japanese, then this should work 100% of the time.

View attachment 26251
View attachment 26252
Click to expand...

\o/ Thanks for the work, Lys!
You're putting it here so readers can MTL by themselves while you slack off, aren't you? I approve.

LysUltima · Jan 14, 2018

Haxagen said: ↑

\o/ Thanks for the work, Lys!
You're putting it here so readers can MTL by themselves while you slack off, aren't you? I approve.
Click to expand...

<.<
It's because I don't want to time deleting these things.
Laziness is all about efficiency.

Osamaru · Jan 14, 2018

(￣▽￣)☕ Tony already beat you to the punch.
https://www.blobtranslations.com/blob-raw-text-cleaner/

LysUltima · Jan 14, 2018

Osamaru said: ↑

(￣▽￣)☕ Tony already beat you to the punch.
https://www.blobtranslations.com/blob-raw-text-cleaner/
Click to expand...

Which is buggy and doesn't work...

yuzuki · Jan 14, 2018

I had been working on a version that was integrated in Google Docs since last week too. XD I think we were all working on one ever since @Tony made his.

View the manual here: https://yuzukicode.blogspot.com/2018/01/google-docs-blobtl-raw-cleaner-add-on.html

Introduction
This is a simple Google Docs add-on that will highlight all lines containing non-English text.

This app works through a whitelist/blacklist strategy. Every character in a line is converted to its UTF-16 code and checked against a whitelist/blacklist. If the character is on the whitelist, all instances of this character are ignored. If the character is on the blacklist, the entire line is marked as "Raw Text". The whitelist takes precedence over the blacklist.

Currently, the blacklist contains the unicode blocks for hiragana, katakana, hangul syllables, and the unicode block for 20,000 of the most common ideographs used in Chinese and Japanese. The unicode blocks containing East-Asian punctuation/symbols is NOT on the blacklist by default. Here are the exact blacklist contents:

Hiragana (unicode block): U+3040..U+309F

Katakana (unicode block): U+30A0..U+30FF

CJK Unified Ideographs (unicode block): U+4E00..U+9FFF

Hangul Syllables (unicode block): U+4E00..U+9FFF

The whitelist is currently empty.

To suggest things to be added to the whitelist/blacklist, you can contact me (@yuzuki on NUF). Alternatively, you can get the source code (which is open-source) yourself and run your own version.

This add-on is still actively being developed, and I'm currently working an in-app menu that will allow you to define your own personal whitelist/blacklist. Suggestions and bug reports are always very welcome.

Installation:
Go to BlobTL Raw Cleaner page in Google Docs add-on store (it's free).

Click the install button and follow the prompts.

The first time I installed it, an empty google doc popped up. If the installation worked properly, you should see something like this:

You can close this empty doc.

Usage:
Go to the google document you are working on.

Go to the Add-ons section and select the BlobTL Raw Cleaner.

I recommend choosing Highlight Raw Text first to check what the script will actually delete.

If you are satisfied with the selection, you can delete the selection with the [Backspace] or [Delete] key on your keyboard.

If you accidentally delete things that you did not want to, you can press Ctrl+Z (undo) to undo the option.

Ignoring lines:
Suppose you have a line that you don't want the script to delete. For instance, a paragraph that contains a kaomoji or some other weird character that isn't in the typical western character set.

You can tell the script to ignore a line by putting two hash tags (##) at the beginning of a line.

Then, the script will ignore that line.

Marking lines for the highlighter:
You may occasionally encounter situations where you want to manually mark lines for highlighting. For example, some lines in the raw may only contain punctuation.

You can manually mark a line for deletion by placing two percent signs (%%) at the start of the line.

Uninstalling:
Go to Add-ons > Manage add-ons...

Then click Manage > Remove

Change log:

5 - Fixed bug with shift+enter - 2018-01-14 14:49

4 - Added a line repairing utility - 2018-01-14 01:21

3 - Fixed a bug in the selector - 2018-01-13 23:32

2 - Added a raw selector - 2018-01-13 22:50

1 - Initial functional version - 2018-01-10 13:54

Credits, Source Code, and License:
The source code for this add-on is freely available here.

MIT License. Please credit Blob Translations if you modify or redistribute it.

Special thanks to @Tony for reviewing the add-on and giving comments, and @BlancFrost for testing. Also, this wouldn't exist if @Action didn't request the feature in the first place.

Finally, you can find similar tools at these following places:

Blob Raw Text Cleaner by Tony

Ebisu Raw Text Remover by LysUltima

yuzuki · Jan 14, 2018

If anybody needs to customize the behavior any further, the source code for the first one I wrote is here:
yuzuki said: ↑
Hi, here's a simple script that will strip all the lines containing raw text (CN/JP/KR) from a google doc. Maybe it will help save some time for the people who are doing it manually.

This post will briefly describe how to do it.

Step 1: Go to your google doc and click on the script editor

Step 2: Copy paste the code into the script editor and press ctrl+S (save it)

Replace anything already in the script editor with this code:
Code:
/**
 * Google Docs Raw Text Cleaner created by @yuzuki of https://www.blobtranslations.com/
 */

/**
 * A custom function that deletes paragraphs containing blacklisted characters.
 */
function cleanRawTranslation() {
  var body = DocumentApp.getActiveDocument().getBody();
  var paras = body.getParagraphs();

  // Loop through each paragraph in the document
  for (var i = 0; i < paras.length; i++) {
    var line = paras[i].getText();

    // Skip and ignore lines that start with: "##"
    if (line.substring(0, 2) == '##') {
      continue;
    }

    // Loop through each character in the paragraph
    for (var j = 0; j < line.length; j++) {
      var c = line.charCodeAt(j);
      if (!isWhiteListed_(c) && isBlackListed_(c)) {
        // Delete this paragraph
        paras[i].removeFromParent();
        break;
      }
    }
  }
}


/**
 * A custom function that checks if a UTF-16 code is whitelisted.
 *
 * @param {Integer} The UTF-16 code to check.
 * @return {Boolean} True if this character is whitelisted.
 */
function isWhiteListed_(c) {
  return false;
}

/**
 * A custom function that checks if a UTF-16 code is blacklisted.
 *
 * @param {Integer} The UTF-16 code to check.
 * @return {Boolean} True if this character is blacklisted.
 */
function isBlackListed_(c) {
  if ((c >= 12352 && c <= 12543) // Hiragana and katakana: U+3040..U+30FF
      || (c >= 19968 && c <= 40959) // CJK Unified Ideographs (Chinese): U+4E00..U+9FFF
      || (c >= 44032 && c <= 55215) // Hangul Syllables: U+AC00..U+D7AF
     ) {
    return true;
  }
  return false;
}
If you like, you can modify the code yourself if you need to adjust its functionality. The script works by looping through every paragraph in the open Google Document, and checking if there are whitelisted or blacklisted characters. If there is a blacklisted character in the paragraph, the entire paragraph is deleted.

The characters are in UTF-16 integers. If you have a character you would like to whitelist/blacklist, you can look up the UTF-16 character code and add it to the appropriate section. You'll need to know how to write an if statement though...

Step 3: Run the script

Push the run button.

Step 4: Give the script permissions to modify your open google doc

If this is your first time running the script on this doc, you will need to grant the script permissions to modify your document. Just follow the on screen prompts.

Most likely you'll get something that says the app isn't verified. This is because I wrote this script in like 30 minutes and didn't submit it to get reviewed by google in their app store yet... maybe I'll do that sometime later (maybe). For now, this script is unverified.

In either case, click on "Advanced" and "Go to [Script Name] (unsafe)".

Choose your google doc account, follow the prompts, and the script will run. Hurrah!

Step 5: Undoing the script

Didn't mean to run the script? No problem. Press control-z and everything the script did will be undone.

Step 6: Manually make the script ignore certain lines

Suppose you have a line that you don't want the script to delete. For instance, a paragraph that contains a kaomoji or some other weird character that isn't in the typical western character set.

You can tell the script to ignore a line by putting two hash tags (##) at the beginning of a line.

Then when you run the script, that line will get ignored.

Hope this helps!
Click to expand...

LysUltima · Jan 14, 2018

yuzuki said: ↑

I had been working on a version that was integrated in Google Docs since last week too. XD I think we were all working on one ever since @Tony made his.

Installation:

Go to BlobTL Raw Cleaner page in Google Docs add-on store (it's free).

Click the install button and follow the prompts.

The first time I installed it, an empty google doc popped up. If the installation worked properly, you should see something like this:

You can close this empty doc.

Usage:

Go to the google document you are working on.

Go to the Add-ons section and select the BlobTL Raw Cleaner.

I recommend choosing Highlight Raw Text first to check what the script will actually delete.

If you are satisfied with the selection, you can delete the selection with the [Backspace] or [Delete] key on your keyboard.

Alternatively, you can choose BlobTL Raw Cleaner > Delete Raw Text, which will do the same thing.

If you accidentally delete things that you did not want to, you can press Ctrl+Z (undo) to undo the option.

Ignoring Lines:

Suppose you have a line that you don't want the script to delete. For instance, a paragraph that contains a kaomoji or some other weird character that isn't in the typical western character set.

You can tell the script to ignore a line by putting two hash tags (##) at the beginning of a line.

Then, the script will ignore that line.

Uninstalling:

Go to Add-ons > Manage add-ons...

Then click Manage > Remove

Troubleshooting:

There is currently one known bug in the add-on.

If you have separated your paragraphs with shift+enter on google docs, google docs does not recognize this mini- line break as a proper new paragraph.

Take a look at this example:

Can you tell the difference between the bad lines and good lines?

For the lines where the English was unintentionally selected, the translator used shift+enter after the Chinese line. This places a line-break without ending the paragraph. Notice how there is no spacing between the paragraphs?

By default, on google docs, if you press the ordinary [enter] key, there should be some extra spacing between paragraphs.

There are two ways to fix this:

(Option A): Copy paste your entire doc to Microsoft Word or some other word processor. Then, copy-paste everything back. If it worked properly, you'll notice that the spacing between the lines was fixed.

(Option B): Run the BlobTL Raw Cleaner > Repair Lines utility.

If you run the highlighter again, everything should be fixed.

Click to expand...

Mine is compatible with shift + enter, so I guess that's a plus

krrizis · Jan 14, 2018

yuzuki said: ↑

If anybody needs to customize the behavior any further, the source code for the first one I wrote is here:
Click to expand...

Brilliant. I remembered asking about this on Discord. So I'm glad this is now live.

Guan Zhong · Jan 14, 2018

lol, OP's did nothing but give me the same text again. Tony's removed the English... = =

EDIT: Wait no, Tony's split it properly, i just didn't notice the English in the upper box. cool

yuzuki said: ↑

I had been working on a version that was integrated in Google Docs since last week too. XD I think we were all working on one ever since @Tony made his.

Installation:

Go to BlobTL Raw Cleaner page in Google Docs add-on store (it's free).

Click the install button and follow the prompts.

The first time I installed it, an empty google doc popped up. If the installation worked properly, you should see something like this:

You can close this empty doc.

Usage:

Go to the google document you are working on.

Go to the Add-ons section and select the BlobTL Raw Cleaner.

I recommend choosing Highlight Raw Text first to check what the script will actually delete.

If you are satisfied with the selection, you can delete the selection with the [Backspace] or [Delete] key on your keyboard.

Alternatively, you can choose BlobTL Raw Cleaner > Delete Raw Text, which will do the same thing.

If you accidentally delete things that you did not want to, you can press Ctrl+Z (undo) to undo the option.

Ignoring Lines:

Suppose you have a line that you don't want the script to delete. For instance, a paragraph that contains a kaomoji or some other weird character that isn't in the typical western character set.

You can tell the script to ignore a line by putting two hash tags (##) at the beginning of a line.

Then, the script will ignore that line.

Uninstalling:

Go to Add-ons > Manage add-ons...

Then click Manage > Remove

Troubleshooting:

There is currently one known bug in the add-on.

If you have separated your paragraphs with shift+enter on google docs, google docs does not recognize this mini- line break as a proper new paragraph.

Take a look at this example:

Can you tell the difference between the bad lines and good lines?

For the lines where the English was unintentionally selected, the translator used shift+enter after the Chinese line. This places a line-break without ending the paragraph. Notice how there is no spacing between the paragraphs?

By default, on google docs, if you press the ordinary [enter] key, there should be some extra spacing between paragraphs.

There are two ways to fix this:

(Option A): Copy paste your entire doc to Microsoft Word or some other word processor. Then, copy-paste everything back. If it worked properly, you'll notice that the spacing between the lines was fixed.

(Option B): Run the BlobTL Raw Cleaner > Repair Lines utility.

If you run the highlighter again, everything should be fixed.

Click to expand...

Nice, thanks for this!

LysUltima · Jan 14, 2018

Guan Zhong said: ↑

lol, OP's did nothing but give me the same text again. Tony's removed the English... = =

EDIT: Wait no, Tony's split it properly, i just didn't notice the English in the upper box. cool
Click to expand...

would you mind giving me the text you inputted that didn't work?

Guan Zhong · Jan 14, 2018

LysUltima said: ↑

would you mind giving me the text you inputted that didn't work?
Click to expand...

Sent a pm.

LysUltima · Jan 14, 2018

Welp, getting the @Guan Zhong's problem with the text I originally used to test it with.

I must've broken something when I was trying to fix something else...

Guan Zhong · Jan 14, 2018

LysUltima said: ↑

Welp, getting the @Guan Zhong's problem with the text I originally used to test it with.

I must've broken something when I was trying to fix something else...
Click to expand...

Been there. Bummer. I'll try her again once she's fixed.

LysUltima · Jan 14, 2018

Guan Zhong said: ↑

lol, OP's did nothing but give me the same text again. Tony's removed the English... = =

EDIT: Wait no, Tony's split it properly, i just didn't notice the English in the upper box. cool

Nice, thanks for this!
Click to expand...

Fixed

LysUltima · Jan 14, 2018

Guan Zhong said: ↑

Been there. Bummer. I'll try her again once she's fixed.
Click to expand...

Wow... nice timing?

Guan Zhong · Jan 14, 2018

LysUltima said: ↑

Wow... nice timing?
Click to expand...

But I'm still getting the same result... hmm..

LysUltima · Jan 14, 2018

Guan Zhong said: ↑

But I'm still getting the same result... hmm..
Click to expand...

It's the same text?
Perhaps refresh?

Guan Zhong · Jan 14, 2018

LysUltima said: ↑

It's the same text?
Perhaps refresh?
Click to expand...

Ah yeah I guess it was a cache issue. Works now. Thanks!

Jimminx · Jan 14, 2018

Thank you! This thing is AWESOME!!

Log in

Resource Raw Text Remover

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Haxagen Fallen's|Addicted to Gacha

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Osamaru 『Shem's Best Pal ✧ Lexi's Ani ✧ Hamster's Keeper』

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

yuzuki [sweet night] [plum blossoms]

yuzuki [sweet night] [plum blossoms]

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

krrizis Runs Ainushi, Read18 & Chaleuria

Guan Zhong Well-Known Member

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Guan Zhong Well-Known Member

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Guan Zhong Well-Known Member

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Guan Zhong Well-Known Member

Attached Files:

Capture.PNG

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Guan Zhong Well-Known Member

Jimminx Well-Known Member

Log in

Resource Raw Text Remover

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Haxagen Fallen's|Addicted to Gacha

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Osamaru 『Shem's Best Pal ✧ Lexi's Ani ✧ Hamster's Keeper』

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

yuzuki [sweet night] [plum blossoms]

yuzuki [sweet night] [plum blossoms]

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

krrizis Runs Ainushi, Read18 & Chaleuria

Guan Zhong Well-Known Member

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Guan Zhong Well-Known Member

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Guan Zhong Well-Known Member

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Guan Zhong Well-Known Member

Attached Files:

Capture.PNG

LysUltima Riichi! Tsumo! Toitoi! Suuankou!?

Guan Zhong Well-Known Member

Jimminx Well-Known Member

Useful Searches