Entai2965 Posted February 13 Share Posted February 13 (edited) [Tool] Sugoi Offline Translator Repackage The following are some repackages I did to strip down Sugoi Offline Translator v4.0 to only the barebones essentials but filled with as many features as possible from the available sources. Disclaimers - Despite the name, the only aspect of Sugoi used in this repackage is the Sugoi offline model v4.0 itself which is subject to a redistribution friendly but non-commercial license by NTT. For more information, see the Licensing section at the bottom. - Most of the information here was shamelessly copy/pasted and then lightly edited information from fairseq, OpenNMT's CTranslate2, MingShiba, and the server authors' readme pages who, all together, wrote a lot of documentation about how everything works and how to use their software. - All credit for the actual hard stuff goes to them. This post just bundles the work they have already done together with some shell scripts to set everything up in a more user-friendly manner. - rant - This forum is anything but user friendly. So apparently this forum preview does not support any sort of sane syntax preview for text->post formatting, just text->formattingbuttons->post, but the forum itself supports some bbc tags even if there is no way to preview that in a sane way before posting. However, URL parsing is inconsistent, [ url ] does not work right, and the /spoiler tag is not always honored leading to reliably broken posts. Adding - or . to empty lines sometimes helps, but why would that be necessary? Just honor the tags! The closing tags must also not be on their own ending line or the parser interprets that as an empty line which is interpeted as "extend the spoiler forever". What programmer decided that? Content in code tags also gets moved to the end of the current spoiler because...? /rant Sugoi Toolkit "Sugoi Toolkit" by MingShiba has several different programs, like Manga Image Upscaler, File Translator, OCR software, and it bundles various utilities to help in manga, LN, and VN translation in general. The latest version at time of writing is the "Sugoi Toolkit" is v8.0 which contains "Sugoi Translator Toolkit v8.0" which contains "Sugoi Offline Translator v4.0". Sugoi Toolkit is free (as in free beer) to download from MingShiba's Patreon About page in the middle of the month, on the ~15-17th, and on https://archive.org/search?query=Sugoi+Toolkit . Website: https://sugoitoolkit.com Patreon: https://www.patreon.com/mingshiba Coffee: https://www.buymeacoffee.com/mingshiba Blog: https://blog.sugoitoolkit.com Demo of the v4.0 model itself: https://blog.sugoitoolkit.com/sugoi-offline-model-v4-0-showcase . Sugoi Offline Translator Sugoi Offline Translator can be used as a translation engine to power Japanese-to-English translations from the clipboard which, when combined with other software like Textractor or OCR software, can translate visual novels in real time. These technologies are included in the Sugoi toolkit. The Offline Translator portion also supports 3rd party applications like Translator++ which can be used to translate arbitrary text, such as contents extracted from visual novels and RPGM games. . The latest version of Sugoi Offline Translator is v4.0 as the the time of writing (early 2024), and it is available in Sugoi Toolkit v6.0, v7.0, and v8.0. . Sugoi Offline Translator v4.0 itself contains - A custom NMT model available under a non-commercial license by "Nippon Telegraph and Telephone Corporation". - A copy of this license is available under apps\Python310\Sugoi.v4.Model.LICENSE. - Meta/Facebook's fairseq project which is used as the inferencing engine for the custom model. - A tiny script to allow the inferencing engine to expose its contents over HTTP. - Some other stuff I do not care about. Repackage Description These repackages are intended to be a minimalistic, high performance, and easy to use alternative to the default Sugoi Offline Toolkit v4.0. They are intended to be replacement-compatible with the Sugoi Toolkit itself but not drop-in compatible meaning that these builds should not be copied into the toolkit but rather used externally of it. The regular functions of the toolkit can also be used if started after the repackaged server software, but this was not tested much since I am mostly concerned with speeding up the batch translations. . The goals of the repackage are to - Support batch translations in Translator++ (public versions only). - Support faster translations than the Sugoi defaults by using GPU acceleration and CTranslate2. - Have smaller, modular, and more easily updatable builds for people who only want the Offline Translator portion of the otherwise very large Sugoi Toolkit. - Create an open source reproducible build environment and instructions to make things more straightforward to update in the future. - It should be possible to rebuild everything from scratch using generic package managers and software such as pip, git, the standard Python installers, and simple open source shell scripts. - All the shell scripts used in this repackage are open source and all projects included in the repackage are open source. - The one exception is Nvidia's CUDA library that is included with PyTorch. For more information see the Licensing section. - Add support for Windows 7 to Sugoi Toolkit for at least as long as until PyTorch no longer supports it. At the time of writing, it still does. - https://pytorch.org/get-started/locally fairseq and CTranslate2 The default configuration in Sugoi Offline Translator v4.0 is to use fairseq + CPU with a flask web server to interface with fairseq. . fairseq is a machine learning and data training library that can also be used to evaluate pretrained models that was originally written by Meta/Facebook and then released under an open source license. . While fairseq works reliably to serve the Sugoi v4.0 model, this combination is very slow and so that is part of why there are so many patches floating around. In addition, the included flask server will not fully release its memory due to OS limitations which is especially problematic when using GPU acceleration. . CTranslate2 is an alternative inference library written by OpenNMT dedicated to serving NMT models efficiently. In terms of CPU processing speed, compactness, and reliability, CTranslate2 is a massive improvement over fairseq. Future versions of Sugoi Offline Toolkit should/might switch to using CTranslate2 as the default inferencing engine because it is hands-down better on the CPU side for end-user inferencing and the Sugoi Toolkit is only intended to be CPU compatible by default. . CTranslate2, does have some drawbacks - If using CUDA, there do not appear to be any particular benefits to CTranslate2 over fairseq since they both rely on PyTorch to implement their CUDA support. - CTranslate2 requires models to be a special format. OpenNMT does have some conversion software to handle that and a script is also included here. - It is not clear how to get DirectML or ROCm to work with CTranslate2. - However, none of OpenNMT's documentation even mentions these technologies. The only reference found to ROCm was this unfixed bug report, https://github.com/OpenNMT/OpenNMT-py/issues/2402 I mean, this may not really be their problem since they mostly use PyTorch for their GPU code anyway, but it would be nice if they acknowledged the existence of AMD graphics card users in some way. PyTorch - PyTorch is the backend engine/library commonly used in Python for machine learning. - PyTorch supports both CPU and GPU inferencing. - With a mountain's worth of caveats, PyTorch is what implements Nvidia CUDA, AMD ROCm, and Microsoft DirectML APIs so that Python 3 applications like fairseq and CTranslate2 can use GPU acceleration on Linux, Windows, and OS/X. - Key limitation is that PyTorch does not natively support OpenCL, a cross platform API for general purpose computing on the GPU. - Both fairseq and CTranslate2 use the PyTorch library to implement their GPU acceleration code. - In other words, getting GPU acceleration in Sugoi is (nearly) synonymous with GPU acceleration in PyTorch. - CTranslate2 has their own C++ code for CPU inferencing that does not use PyTorch, which is why it is very fast and also why CTranslate2 + CPU does not require PyTorch at all. - In contrast, fairseq always needs PyTorch, including for just fairseq + CPU. - In other words, the only reason to use PyTorch is for GPU acceleration. fairseq is not needed at all, except if using DirectML since PyTorch + ROCm on native Windows is not currently supported. AMD graphics cards - Glossing over the details, ROCm is basically AMD's version of CUDA that works on both Linux and Windows. However, there are no ROCm + PyTorch + Windows builds available as of early 2024. There are some reports of getting ROCm to work under Linux with PyTorch. - Anecdote. https://news.ycombinator.com/item?id=37795443 - DirectML is closed source GPU acceleration based on Microsoft's DirectX 12 API and is available on Windows 10 1703+/1903+ including the Windows Subsystem for Linux. This is a promising alternative for use with AMD graphics cards but this is still in very early development, extremely finicky, and extremely slow, often slower than CTranslate2 + CPU. - Note that DirectML is the only GPU acceleration available for Windows + AMD graphics cards that does not require writing raw code for fairseq and CTranslate2 because PyTorch does not support OpenCL or ROCm + Windows and because oneDNN is not supported in CTranslate2. - ROCm does work on WSL 2, but none of the server authors have documentation on getting that to work. - oneDNN https://github.com/oneapi-src/oneDNN also supports ROCm, so adding support AMD GPU support to CTranslate2 via oneDNN + ROCm looks promising, but this is not under active development. There was some talk about integrating ROCm via Intel's oneDNN library in their issues page. - DirectML and ROCm are very new technologies under active development. Neither ROCm, nor DirectML have any widespread configuration that works reliably with modern PyTorch, a core Python machine learning library used by both fairseq and CTranslate2. - These technologies are very dependent on having this particular OS version, multiple obscure software versions that conflict with other software, a specific $400 graphics card, etc. The odds of being able to tick all of those boxes all in a row is very low right now. - This situation for AMD graphics card users should improve in the future since both of these technologies show a lot of promise and are under active development by both AMD and Microsoft. Microsoft DirectML - Okay so, this is more of a meme and a feature checkbox for software developers as opposed to something genuinely useful in the real world. Do not use this. Why not? Well... - When DirectML is faster than CTranslate2 CPU, DirectML is only marginally (10-30%) faster. - When CTranslate2 CPU is faster than DirectML, CTranslate2 can be 2.5x+ faster. - It uses too many old and special versions of Python, PyTorch, or fairseq. The DirectML repackage below cannot be updated because updating any of the versions for the major components breaks all of it. - The Microsoft PyTorch-DirectML library has not been updated in almost a year. Considering how fast technology moves, especially machine learning and AI right now, this is inexcusable. It might as well be abandonware at this point. - Microsoft is focusing their efforts on their main closed source DirectML library. Until they get around to porting that updated code to their PyTorch version, this older software is just not worth using. - DirectML in particular cannot be installed alongside the latest PyTorch v2.2.1 or newer, so that is better off running in an isolated environment where it is free to use its old 2.0 version without conflicting with the global PyTorch configuration. - If you really want to run everything from one global PyTorch version, then use Python 3.10 with PyTorch 2.0.1 CUDA 11.8. While that should work and instructions are at https://pytorch.org/get-started/previous-versions , I am not supporting that. - Running Python 3.8 or 3.9 + DirectML requires building an old fairseq version from source. You have been warned of this. - Please read this notice of Microsoft's included spyware in their DirectML library https://pypi.org/project/torch-directml and their Privacy Statement https://go.microsoft.com/fwlink/?LinkID=824704 . - Summary is that you have no expectation of privacy while using anything written by Microsoft. - So, why include this then? Because this a meme and a feature checkbox for software developers, so it felt like a waste not to include it even if really only useful to compare vs PyTorch CUDA and CTranslate2 CPU. HTTP servers available for fairseq/CTranslate2 - OpenNMT's native implementation of a simple rest server. Includes support for their CTranslate2 repository and other model formats. - https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/bin/server.py - OpenNMT's implementation is the only decently supported one here, but OpenNMT is not very focused on end-user friendly distribution or documentation. Their 'end-user' is other programmers. - reAlpha39's restserver for fairseq. This only works for PyTorch 2.0? and a different model format than Sugoi uses maybe? I was unable to get this working. - https://github.com/reAlpha39/fairseq-translate-server/blob/main/rest_server.py - https://git.sr.ht/~asdf/sugee/tree/main/item/sugee/server.py - gdiaz384's py3translationServer. Works for fairseq + CTranslate2 using CPU + CUDA + DirectML. - https://github.com/gdiaz384/py3translationServer - Lots of documentation. - Default Sugoi flask + fairseq server. - Available at https://github.com/cbc02009/vno-local-cpu/blob/main/flaskServer.py - The default is fairseq + CPU only. flask + CTranslate2 server is availalble on Sugoi's discord along with GPU patches. The flask scripts for fairseq, CTranslate2, CPU, and GPU are available as completely different scripts, and there are patches to Sugoi Offline Translator available on top of that which makes everything unnecessarily confusing since their documentation is just awful. Notes on the repackages (Windows only) - Linux users, read the "Linux users" section. - All of these are 'unofficial' and hence not supported by MingShiba. If something does not work, please do not bother them about these builds. - The usual comments about no warranties or guranteed support apply here as well. See licencing section for details. - These repackage.7z files are NOT intended to be drop-in replacements for Sugoi Offline Translator v4.0. That would require messing with Sugoi's internal scripts and that is not happening. - It may be possible to merge these with the toolkit, but I am not supporting that. - However, they do -intentionally- conflict with Sugoi's offline server. - The idea is to allow for external integration by launching the server included in this repackage and then launch Sugoi offline translator after it from the electron menu GUI. - That should bring up the translation-from-clipboard UI box without having to modify any of the messy Sugoi internals and the clipboard translations should still work. - All of the software installs and runs in "portable" mode. There are no changes are ever made to the local system at all. It should never trigger a default settings UAC prompt. To uninstall the software, delete the folder. - However, there is an automatic prompt by the Windows firewall at the end asking to allow the Sugoi model to be served over the network or to keep everything local to the current system. - If you want to translate stuff on the same computer you are running the server, then accessing http://localhost does not need firewall permissions at all. Just deny the request. - If you would like to have the translation server always running and translate stuff on another computer on the same network, then allow the firewall exception. - If you do not understand what the above three sentences are trying to explain, deny the firewall exception. - The portable local environment itself consists of untouched portable instances of Python 3.10, git, and some small .bat scripts. - These are optional and only included for convenience. If you have your own versions of Python 3.10 and git installed, then there is no particular benefit to using the provided ones at all. Just delete them and update the paths in the .bat to point to your python.exe and git.exe - The following variables at the top of the Sugoi repackage scripts should point to your versions: gitExe, pythonExe, pythonHomeDir. - With some caveats, any existing Python 3.8-3.12 installation should work including both portable or global versions because PyTorch v2.2.1 added support for Python 3.12. - For my own sanity, I only support Python 3.10, the version included in the repackages, because things can break when switching Python versions. Especially for Python <= 3.7, you are on your own since a lot of the libraries used in this project no longer support it, including fairseq. Incidently, Python <= 3.7 is considered end of life. - The fairseq/CTranslate2.bat scripts will need to point to your locally installed Python. - ROCm does not have a repackage because PyTorch does not have any ROCm+Windows builds as of early 2024 and also because CTranslate2 does not support ROCm natively or through oneDNN. - Basically, ROCm + Windows just does not work because the software to support it has not been written yet. - When will it be written? Consider learning programming and writing it yourself or figuring out how to use it on WSL 2 and then detailing to the rest of us how you managed to make it work. - By default, most of the repackages do not include the model or PyTorch. - On first run, PyTorch will be downloaded and installed in an isolated and portable version of Python 3 for all versions except Sugoi-Repackage-CPU, which does not use PyTorch at all. - PyTorch v2.2.1 CUDA 11.8 was chosen for compatibility reasons and should almost always work on GTX 600 series Nvidia cards and newer. - For Windows 7 SP1, there is a bundled fix to downgrade CUDA 11.8 -> 11.4. For details, read "Windows 7 and 8" below. - This fix only attempts to work for Windows 7 SP1. Pre-SP1 Windows 7 and Windows 8 users, please apply the fix manually if needed. The fix will not be attempted on Windows 10 or 11 systems. - The fix relies on Powershell and the .Net Framework 4.5+. If those are not installed, then apply the fix manually. - If these limitations apply, then please update Powershell with KB3191566 and .Net Framework to 4.8. - For PyTorch CUDA 12.x, install PyTorch using apps\scripts\installPyTorch-CUDA-12.bat before launching the main Sugoi-Repackage.reproducible.bat. - py3fairseqTranslationServer was renamed to py3translationServer because why? So I removed git and added it directly so the server author cannot break these repackages again. However, that means that these repackages will not automatically incorproprate updates from now on. Update the server software manually if this is important. - The Sugoi-Repackage.reproducible.bat files will move themselves to apps\scripts\ for safe-keeping once they have finished setup. To review the installation help they present, just open them as text files and scroll down. - This repackage should also be compatible with the original models at JParaCrawl http://www.kecl.ntt.co.jp/icl/lirg/jparacrawl and with Facebook/Meta's pretrained fairseq translation models https://github.com/facebookresearch/fairseq/blob/main/examples/translation/README.md - If 7-Zip says "Unsupported compression method" while extracting these archives, update 7-Zip https://7-zip.org to a more recent version, or use WinRar. Converting to the CTranslate2 model format - If using CTranslate2, then it needs the model converted to a specific format. - For Sugoi, there is a conversion script to convert the Sugoi Offline v4 model to the CTranslate2 format included at apps\scripts\fairseqToCTranslate2_converter.Sugoi.bat - Using the conversion script requires a previously downloaded version of Sugoi Toolkit, but it means the model in CTranslate2 format does not have to be downloaded, limiting the use of bandwidth which is important for Sugoi-Repackage-CPU, options B, and Sugoi-Repackage-Global, option D. - The conversion script is known to be compatible with Sugoi Toolkit versions 4.0, 6.0, and 7.0. - To convert the default Sugoi model to CTranslate2 format, copy apps\scripts\fairseqToCTranslate2_converter.Sugoi.bat to - Sugoi-Toolkit\Sugoi_Translator_Toolkit\Code\Power-Source\Python39 - Double click on it and it will generate the model in the sugoi_v4model_ctranslate2 folder. - If moving a file from one place to another and double clicking on it is too complicated, then download the already converted model instead. - To convert other models to CTranslate2 format, refer to OpenNMT's documentation https://opennmt.net/CTranslate2/conversion.html - Their conversion software gets installed under Python310\Scripts\* when doing pip install ctranslate2 - For fairseq, do python.exe ct2-fairseq-converter.exe --help - Windows 7 users will need an updated api-ms-win-core-path-l1-1-0.dll See Windows 7 and 8. Pick one of the following. - DL https://mega.nz/folder/ZORBTLAS#f2G4ih7crgWIN-_uSjbqAA - The different repackage versions have - Support for the Sugoi v4.0 offline model. - Support for the fairseq and CTranslate2 inference libraries. This includes CPU, CUDA, and/or DirectML support. - Support for py3translationServer since their documentation is the most readable. - A preconfigured batch script to configure an isolated local environment. Repackage Option A) Sugoi-Repackage-CUDA.7z - Description - Normal and most flexible version recommended for most users. Supports both fairseq and CTranslate2 using both CPU and CUDA 11 or 12. - Has a reasonable download+install time. - Does not include PyTorch or the Sugoi model. - Sugoi-Repackage-CUDA-Full.7z is the same as "Sugoi-Repackage-CUDA.7z", but comes bundled with PyTorch CUDA 11.8 and the other dependencies which bloats the compressed size from <100MB to ~2.48 GB. Since setup has been done already and it includes the both versions of the models. Sugoi-Repackage-CUDA installation instructions 1. Download Sugoi-Repackage-CUDA.7z - Size is 32.1 MB (archive) -> 160 MB (extracted) -> ~5.72 GB (post PyTorch v2.2.1 CUDA 11.8 install, not including model) 2. Extract the archive. Right-click->7-Zip->Extract here. - If 7-Zip says unsupported compression method, then delete any extracted contents, update 7-Zip, and try extracting again. 3. By default, Sugoi-Repackage-CUDA.bat will install PyTorch v2.2.1 CUDA 11.8. - To install PyTorch v2.2.1 CUDA 11.8, run ~Sugoi-Repackage-CUDA.bat - To install PyTorch v2.2.1 CUDA 12.1 instead - copy ~apps\scripts\installPyTorch-CUDA-12.bat to apps\Python310 - After copying it to the above folder, double click on it. - After installing PyTorch CUDA 12, run ~Sugoi-Repackage-CUDA.bat - To install PyTorch v2.2.1 CPU, copy ~apps\scripts\installPyTorch-CPU.bat - copy ~apps\scripts\installPyTorch-CPU.bat to apps\Python310 - After copying it to the above folder, double click on it. - After installing PyTorch CPU, run ~Sugoi-Repackage-CUDA.bat - To install newer versions of PyTorch, use the "latest" versions of the scripts instead of the "v2.2.1" versions. 4. Downloading and installing PyTorch and the dependencies will take a while. While that is running, decide to use either fairseq or CTranslate2. - If Sugoi Toolkit is not available on the local system, then download the model using the links provided below. - fairseq - The fairseq model should either be placed at "sugoi_v4model_fairseq" such that the following files exist. - Sugoi-Repackage-CUDA\sugoi_v4model_fairseq\big.pretrain.pt - Sugoi-Repackage-CUDA\sugoi_v4model_fairseq\spm\spm.ja.nopretok.model - Sugoi-Repackage-CUDA\sugoi_v4model_fairseq\spm\spm.ja.nopretok.vocab - Sugoi-Repackage-CUDA\sugoi_v4model_fairseq\spm\spm.en.nopretok.model - Sugoi-Repackage-CUDA\sugoi_v4model_fairseq\spm\spm.en.nopretok.vocab - Alternatively, update the "set modelPath=" line in the following script. - py3translationServer.py.fairseq.CUDA.bat - CTranslate2: - If Sugoi Toolkit has already been downloaded, then convert the model with the procedure found in the "Converting to the CTranslate2 model format" section. - Otherwise, download the model using the links provided below. - The model should be placed at sugoi_v4model_ctranslate2 such that the following files exist. - Sugoi-Repackage-CUDA\sugoi_v4model_ctranslate2\model.bin - Sugoi-Repackage-CUDA\sugoi_v4model_ctranslate2\spm\spm.ja.nopretok.model - Sugoi-Repackage-CUDA\sugoi_v4model_ctranslate2\spm\spm.ja.nopretok.vocab - Sugoi-Repackage-CUDA\sugoi_v4model_ctranslate2\spm\spm.en.nopretok.model - Sugoi-Repackage-CUDA\sugoi_v4model_ctranslate2\spm\spm.en.nopretok.vocab - Alternatively, update the "set modelPath=" line in the following scripts. - py3translationServer.py.CTranslate2.CPU.bat - py3translationServer.py.CTranslate2.CUDA.bat 5. Once the model has been moved to the appropriate directory and PyTorch has finished installing, the server can be run by double clicking on any of the following files. - py3translationServer.py.CTranslate2.CPU.bat - py3translationServer.py.CTranslate2.CUDA.bat - py3translationServer.py.fariseq.CUDA.bat - Only use one at a time. - The above files launch py3translationServer.py based upon a template provided with the .py. Sugoi-Repackage-CUDA-Full installation instructions 1. Download Sugoi-Repackage-CUDA-Full.7z - Size: ~2.48 GB (archive) -> 5.72 GB (post PyTorch v2.2.1 CUDA 11.8 install, not including models). Models add 1 GB/model. 2. Extract the archive. Right-click->7-Zip->Extract here. - If 7-Zip says unsupported compression method, then delete any extracted contents, update 7-Zip, and try extracting again. 3. Wait a long time for it to extract and then run Sugoi-Repackage-CUDA.bat 4. Decide to use either fairseq or CTranslate2. 5. The server can be run by double clicking on any of the following files: - py3translationServer.py.fairseq.CUDA.bat - py3translationServer.py.CTranslate2.CPU.bat - py3translationServer.py.CTranslate2.CUDA.bat - Only use one at a time. - The above files launch py3translationServer.py based upon a template provided with the .py. Repackage Option B) Sugoi-Repackage-CPU.7z - Description - This version is for the bandwidth and GPU impaired that know they will only be using CPU inferencing. It supports only CTranslate2 + CPU. - If you know your GPU will not work with CUDA 11.x, then this is the best option. - Has the fastest download+install time. - Does not need PyTorch at all, thus saving lots of space and bandwidth! - The model is not included. - The model must be in CTranslate2 format. Sugoi-Repackage-CPU Installation instructions: 1. Download Sugoi-Repackage-CPU.7z - Size: 32.1 MB (archive) -> 160 MB (extracted) -> ~527 MB (post install, not including model) 2. Extract Sugoi-Repackage-CPU.7z - If 7-Zip says unsupported compression method, then delete any extracted contents, update 7-Zip, and try extracting again. 3. Run Sugoi-Repackage-CPU.bat - By default, Sugoi-Repackage-CPU.bat will not install PyTorch or fairseq. 4. Download or create the sugoi_v4model in CTranslate2 format. - If Sugoi Toolkit is available on the local system, then convert the model with the procedure found in the "Converting to the CTranslate2 model format" section. - If Sugoi Toolkit is not available on the local system, then download the CTranslate2 model using the links provided below. - The CTranslate2 model should be placed at "sugoi_v4model_ctranslate2" - Alternatively, update the "set modelPath=" line in the following script: - py3translationServer.py.CTranslate2.CPU.bat 5. Add the CTranslate2 model to sugoi_v4model_ctranslate2 such that the following files exist. - Sugoi-Repackage-CPU\sugoi_v4model_ctranslate2\model.bin - Sugoi-Repackage-CPU\sugoi_v4model_ctranslate2\spm\spm.ja.nopretok.model - Sugoi-Repackage-CPU\sugoi_v4model_ctranslate2\spm\spm.ja.nopretok.vocab - Sugoi-Repackage-CPU\sugoi_v4model_ctranslate2\spm\spm.en.nopretok.model - Sugoi-Repackage-CPU\sugoi_v4model_ctranslate2\spm\spm.en.nopretok.vocab 5. Once the model has been moved to the appropriate directory, the server can be run by double clicking on - py3translationServer.py.CTranslate2.CPU.bat - The above file just launches py3translationServer.py based upon a template provided with the .py. Repackage Option C) Sugoi-Repackage-DirectML.7z - Description - This version is for gamblers who want to roll the dice with Microsoft DirectML on Windows 10 1903+. - Read "DirectML" above and then use something else. - Does not include PyTorch or the model. - Do not expect anything to avoid the inevitable disapointment. Only sometimes works with fairseq ...for now. - This PyTorch-DirectML repackage is provided as-is. No support is provided here if it does not work or breaks later. - If it does not work, I do not care. If Microsoft does not care about supporting their library, then why should I? Sugoi-Repackage-DirectML Install Guide: 1. Download Sugoi-Repackage-DirectML.7z 2. Extract the archive. Right-click->7-Zip->Extract here. - If 7-Zip says unsupported compression method, then delete any extracted contents, update 7-Zip, and try extracting again. 3. After extraction, run Sugoi-Repackage-DirectML.reproducable.bat 4. Decide to use either fairseq DirectML or CTranslate2 CPU. - If Sugoi Toolkit is not available on the local system, then download the model using the links provided below. - fairseq - The fairseq model should either be placed at "sugoi_v4model_fairseq" such that the following files exist. - Sugoi-Repackage-DirectML\sugoi_v4model_fairseq\model.bin - Sugoi-Repackage-DirectML\sugoi_v4model_fairseq\spm\spm.ja.nopretok.model - Sugoi-Repackage-DirectML\sugoi_v4model_fairseq\spm\spm.ja.nopretok.vocab - Sugoi-Repackage-DirectML\sugoi_v4model_fairseq\spm\spm.en.nopretok.model - Sugoi-Repackage-DirectML\sugoi_v4model_fairseq\spm\spm.en.nopretok.vocab - Alternatively, the "set modelPath=" line should be updated in the following script. - py3translationServer.py.fairseq.DirectML.bat - CTranslate2: - If Sugoi Toolkit has already been downloaded, then convert the model with the procedure found in the "Converting to the CTranslate2 model format" section. - Otherwise, download the model using the links provided below. - The model should be placed at "sugoi_v4model_ctranslate2" such that the following files exist. - Sugoi-Repackage-DirectML\sugoi_v4model_ctranslate2\model.bin - Sugoi-Repackage-DirectML\sugoi_v4model_ctranslate2\spm\spm.ja.nopretok.model - Sugoi-Repackage-DirectML\sugoi_v4model_ctranslate2\spm\spm.ja.nopretok.vocab - Sugoi-Repackage-DirectML\sugoi_v4model_ctranslate2\spm\spm.en.nopretok.model - Sugoi-Repackage-DirectML\sugoi_v4model_ctranslate2\spm\spm.en.nopretok.vocab - Alternatively, the "set modelPath=" line should be updated in script. - py3translationServer.py.CTranslate2.CPU.bat 5. Once the model has been moved to the appropriate directory and PyTorch has finished installing, the server can be run by double clicking on any of the following files: - py3translationServer.py.fairseq.DirectML.bat - py3translationServer.py.CTranslate2.CPU.bat - Only use one at a time. - The above files launch py3translationServer.py based upon a template provided with the .py. Repackage Option D) Sugoi-Repackage-Global - Description - These are the same scripts as above but slightly tweaked to use the global git and Python interpreter. I have not tested these much. - This does not contain any binary files, so it is the smallest but requires a working local environment which is your responsibility to set up properly and hence not supported by me. - Since it requires setting everything up manually, this is actually slower to set up than just using a portable local environment. However, if running different software that relies on PyTorch already on the same system, like Stable Diffusion, then it can save a lot of space to take the time to merge the environments manually if it can somehow all be made to work together. - The scripts assume git is available and a Python 3.10 environment. SD-webui also uses Python 3.10. - If running a different Python version, then update the fairseq install link from py3translationServer's readme. <-Important - These also require manually changing the "set pythonExe=" and "set pythonHomeDir=" lines which is necessary to prevent any accidental and potentially conflicting changes to the local system. - The scripts will not run if these paths are not updated. - If you do not understand how to do this, then use one of the non-global options instead. Then, if anything goes wrong, you can just delete the folder and no changes are ever made to your local system. - These might require an administrative command prompt to work due to needing to make changes to Python's site-packages folder, which might have been installed system wide back when Python was installed. If Python was installed as only the local user, then an administrative command prompt should not be necessary. - To install, adjust the paths in the scripts, and install using the same instructions as above with the chosen script. - Be sure to move any unused scripts to apps\scripts\ for safe-keeping. Sugoi Model Links This is the same model included in Sugoi Toolkit v6 and v7. There is no need to download it again if the Sugoi Toolkit is already on the local system. The model is under: Sugoi Toolkit\Sugoi-Translator-Toolkit\Code\backendServer\Program-Backend\Sugoi-Japanese-Translator\offlineTranslation\fairseq\japaneseModel and the related sentencepiece models are at [...]\offlineTranslation\fairseq\spmModels Copy it or update the scripts to point to it. To convert it for use with CTranslate2, follow the instructions under "Converting to the CTranslate2 model format". Alternatively, here are some links. . sugoi_v4model_fairseq.7z Mega: https://mega.nz/file/hPwmSSKS#QZ7k3HfgScDnLx3t-e0F3SWC1OHbaTKnVMeTV-upHV8 Pixeldrain: https://pixeldrain.com/u/XmwgGyNf Mediafire: https://www.mediafire.com/file/53lc5brfgi59skz Size: 584 MB -> 1.02 GB CRC32: 63B4D51A CRC64: 1BD06D682CB47BEF SHA256: 23df673fb373eb6efee122181ba697f72f62edfe11178b55330abe53064564ad SHA1: 8310235ea88827f4af7c7d2abceaae7aeb099869 . Name: sugoi_v4model_ctranslate2.7z Mega: https://mega.nz/file/Bfw0lRiS#7gpA_8uXOoyCECybYcGqe8sItEHCNqPV29qy8N_iKFM Pixeldrain: https://pixeldrain.com/u/qh7oUVnj Mediafire: https://www.mediafire.com/file/em2bb4bmi9xdtj2 Size: 593 MiB -> 1.03 GB CRC32: 55CA26DA CRC64: C062C1AE64160010 SHA256: 8556841ffbe12c600fab11171da86badcaccd6f977f4e2f9260802bff526c5fd SHA1: d0cab759e465303f9cf4b6252dac99b33dcabd32 . More hashes Sugoi Toolkit v6.0; Sugoi Offline Translator v4.0 model hashes big.pretrain.pt Size: 1.02 GB CRC32: 827C5EC5 CRC64: B31AC4A3D723D863 SHA256: 59bcb37b859a6ae2070c3aa7683c6968badc532e667f19bb6d8690553e38a18d SHA1: 84a24e902ef8943e1ab208379c7f9300d2afa105 . Sugoi Toolkit v7.0; Sugoi Offline Translator v4.0 model hashes big.pretrain.pt Size: 1.02 GB CRC32: 827C5EC5 CRC64: B31AC4A3D723D863 SHA256: 59bcb37b859a6ae2070c3aa7683c6968badc532e667f19bb6d8690553e38a18d SHA1: 84a24e902ef8943e1ab208379c7f9300d2afa105 . model.bin (Converted model to CTranslate2 format) Updated PyTorch to 2.2.0 CUDA 11.8 and CTranslate2 to 3.24.0 prior to conversion. Size: 1.02 GB CRC32: 959769F1 CRC64: 8DC0DFDD9D4EC597 SHA256: 48db52377c6f3fef10b957e416def0160b7effbca7ff1dc81bd34d71a2fee063 SHA1: 2b5e54cdbd303e725402c10d6bf2f5388991dd0f . spm.ja.nopretok.model: CRC32: 91A3B04C CRC64: 407B43807F9F5050 SHA256: 1bff3529a8e0bd898f00707a4e36dc16540d84112cc8a4a14462c0099e4aab9d SHA1: eb3e9251e8b0ce6337f47c180048289afee013d4 . spm.ja.nopretok.vocab: CRC32: 1FF95F1E CRC64: 70AF3FE96F7639A7 SHA256: 668ac7a2a3b34e2264cfb77d5c4a28d7b3919a2103c3679a4fccc150eba4533d SHA1: 507ba6c42052598843514c48f76fe81e57144e3f . spm.en.nopretok.model: CRC32: 13550ADC CRC64: 27DE9D04944A25E3 SHA256: 183aad11f5fa48b21fdbbb7a97082d160b86cbcc4f9dc5e61d0eebd48390d36a SHA1: ca4c3e447e15fea505677591d75431ca5b3f570a . spm.en.nopretok.vocab: CRC32: 94F0ADD0 CRC64: D09A270EA509D77E SHA256: 08931d4c67c6d19f23a3c128d9997d0c09edb4566c90d440f5492b92f42694a2 SHA1: 2343d7382f59a73fa4dc8ee84c0bd4abd06eaaa2 Linux users For CPU inferencing, use - pip install ctranslate2 - If that does not work, try pip3 install ctranslate2 - The CTranslate2 version of the Sugoi model: sugoi_v4model_ctranslate2 - One of the HTTP servers above. - Python 3.8+ is required for the latest PyTorch and CTranslate2 versions. Check using - python --version - If that does not work right, then try: - python3 --version . For GPU inferencing First, focus on getting PyTorch to work with your local Python config. Website: https://pytorch.org/get-started/locally/#linux-prerequisites If applicable, update the OS driver for your GPU. The host OS and video card driver must expose the correct CUDA/ROCm version for software to be able to use those APIs. Then, install the PyTorch package that supports CUDA/ROCm for the installed GPU. Once that is done, check to make sure PyTorch works: . --code-- python import torch x = torch.rand(5, 3) print(x) ctrl+Z --code-- . Once PyTorch works, check to make sure GPU acceleration also works with the following commands: . --code-- python import torch torch.cuda.is_available() --code-- . Note that torch.cuda.is_available() is the offical method for checking to see if the GPU (generic) is accessible from PyTorch, not just for GPUs that support CUDA. See AMD's documentation. https://rocm.docs.amd.com/projects/install-on-linux/en/develop/how-to/3rd-party/pytorch-install.html#testing-the-pytorch-installation Once PyTorch + GPU acceleration works, install at least one of the inference engines: . --code-- pip install fairseq pip install ctranslate2 --code-- - Then either write some code to link fairseq or Ctranslate2 to the model or use one of the different HTTP servers above. py3fairseqServer, the one used in this repackage, requires . pip install tornado sentencepiece psutil . On Linux, fairseq2 might also be worth looking at. It is a Linux only successor to fairseq that is focused on LLMs. Windows 7 and 8 - To run Python 3.9+ on Windows 7, and OpenNMT's fairseq->CTranslate2 formatting tool with Sugoi, use an updated version of api-ms-win-core-path-l1-1-0.dll - https://github.com/nalexandru/api-ms-win-core-path-HACK - Place at Sugoi_Translator_Toolkit_V7\Code\Power-Source\Python39 or install globally. - An alternative is to use these installers which bundle the above file: https://github.com/adang1345/PythonWin7/ - For CPU inferencing, use CTranslate2 via the Sugoi-Repackage-CPU. See above for instructions. - For GPU inferencing, only Nvidia GPUs are supported. - Nvidia did not make any Windows 7 drivers for their RTX 4000 series cards, so only RTX 3000 series and older are supported. - Nvidia did not make any Windows 8 drivers for their RTX 4000 series, RTX 3000 series, RTX 2000 series, GTX 1600 series cards, so only GTX 1000 series and older are supported. - AMD/Intel GPUs + DirectML is not available because DirectX 12 is not available on Windows 7, 8, and old versions of Windows 10. - AMD GPUs with ROCm would require PyTorch integration, but there are no Windows builds at all for ROCm + PyTorch as of early 2024. . Prerequisites: - Make sure the Nvidia GPU is recent enough. Must have Compute Capability 3.5+ https://developer.nvidia.com/cuda-gpus - Make sure the WHQL driver is recent enough. CUDA 11.4 is the last supported CUDA version for Windows 7. For CUDA 11.4, the driver version must be 365+. - Nvidia Control Panel->Help->System Information->Driver version. - If it is too old, then update the driver https://www.nvidia.com/download/find.aspx Use WHQL drivers only. . Installation: - Install PyTorch CUDA 11.8 normally as per the table at https://pytorch.org while using the local Python environment. - Alternatively, use the included script at ~apps\scripts\installPyTorch-CUDA-11.bat which does the same thing. - If using the script, copy it to "apps\Python10\" first, and then double click on it. - After installing PyTorch CUDA 11.8, downgrade the version of CUDA included to 11.4. - Use the cudart64_11.4 available here: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7379 - Note: This file is also available from Nvidia directly as well but requires an 800 MB download. - Nvidia also encourages people who want to download the cuDNN runtime to sign up for a developer account. - https://developer.download.nvidia.com/compute/redist/cudnn/v8.2.4/cudnn-11.4-windows-x64-v8.2.4.15.zip - Put it at Python\Lib\site-packages\torch\lib. - Rename the existing one so it has a different extension or delete it. - Sugoi-Repackage-CUDA, option A, attempts to automatically apply this fix for Windows 7 SP1 users when installing CUDA 11. Apply it manually if the automation fails. - If using Sugoi-Repackage-CUDA-Full, then apply the patch manually. not so Frequently Asked Questions What is this for? Considering this is Fuwanovel, the idea is to help translate novels, specifically visual novels, specifically Japanese-only visual novels. This software helps non-Japanese speakers read Japanese-only VNs by means of computer assistance. This opens up the potential readers of Japanese-only VNs to the rest of the planet. That should directly lead to more sales of those Japanese-only novels, leading to the developers seeing a rise in interest and hopefully publishing even more of them. Everybody wins. . Since this is machine learning software (an NMT specifically), it can be used to either translate untranslated novels in real time by using Agent https://github.com/0xDC00/agent or Textextractor https://github.com/Artikash/Textractor or as one of many options when using software like Translator++ that aggregates many types of translations allowing you to pick the best one, so when you read your novel again, it is more comprehensible than pure MTL. . How to actually do either of those things as a step-by-step guide is slightly outside the scope of this post/thread. This post just focuses on making the core engine that actually does the actual translation easier to use, and also making it super fast. . NMTs? What? For more technical documentation focused on the NMT technology itself as used by Sugoi read Wikipedia. The successor or logical progression of NMT technology is in Large Language Models (LLMs) which is synonymous with AI. https://en.wikipedia.org/wiki/Neural_machine_translation https://en.wikipedia.org/wiki/Large_language_model https://en.wikipedia.org/wiki/Generative_artificial_intelligence . Is there a UI? Open a web browser to http://localhost:8501 to connect to it. . How to change the translation engine, CPU/GPU mode, and other settings. Right click on the .bat file. Select Edit. Change the appropriate setting. Read the documentation. https://github.com/gdiaz384/py3translationServer#parameters . To change the translation engine and model: 1. Find the line that says: set mode= 1. Change that to fairseq or ctranslate2. Example: --code-- set mode=fairseq set mode=ctranslate2 --code-- fairseq and ctranslate2 use different model formats, so the modelPath= must be changed as well. For fairseq, the model name must be specified. set modelPath=sugoi_v4model\big.pretrain.pt For CTranslate2, just the folder is enough. set modelPath=sugoi_v4model_ctranslate2 . Preloading the model - py3translationServer launches in multiprocess mode by default to ensure proper memory management. - It creates a subprocess for the inferencing engine and model by default. - This behavior can be disabled by using `--preloadModel` (`-pm`). - Multiprocess mode does not use system resources, like VRAM, when the translation engine is not in use and instead loads the model whenever it is needed. - Multi vs one process have the same processing speed, the time it takes to process the data. The difference is how often the model will be loaded. It can either be loaded once and always kept in memory consuming resources, or multiple times, increasing output latency by a flat amount, but only using memory when the model is invoked. . Translator++ integration - For Translator++ under Options->Sugoi Translator, remember to enable 'Send requests in batch' and then set 'Max Parallel job' as high as possible. - Higher batch sizes results in significantly faster processing because then Translator++ does not have to invoke the translation engine multiple times and the translation engine can process everything all at once. - However, this results in increased RAM/VRAM consumption because the translation engine has to hold all requests in memory simultaneously. - ~500 is a decent start for most desktop systems to allow measuring max memory consumption to see how the local system handles it. Raise or lower as needed. - Try ~100 for GPUs with a low amount of VRAM, like <= 4GB, and raise or lower as needed. . The "program crashed successfully" immediately after launch - Only one program can listen on the same port at any one time . Check to make sure python.exe is not already running in the background. Note that Sugoi's server will try to listen on the same port and so can cause this error too. Licensing - Microsoft DirectML is not open source and has two licenses. - https://github.com/microsoft/DirectML/blob/master/LICENSE - https://www.nuget.org/packages/Microsoft.AI.DirectML/1.13.0/License - Since DirectML is not distributed in this repackage, this repackage is immune to their terms. - However, DirectML users should still be aware of Microsoft's Data Collection Notice and their Privacy Policy https://pypi.org/project/torch-directml - PyTorch.org bundles the CUDA runtime with their PyTorch software that is under the license granted to them by Nvidia. That license allows developers to bundle the CUDA runtime with software that utilizes it, which is also what is being done here *only* in the Sugoi-CUDA-Full.7z repackage. None of the other repackages are subject to this restriction. - The Sugoi offline v4 model is under a non-commercial license by Nippon Telegraph and Telephone Corporation (NTT). See their license on JParaCrawl's site of which Sugoi model is derivative of I am assuming https://www.kecl.ntt.co.jp/icl/lirg/jparacrawl - A copy of this license is included with the models and at apps\Sugoi.v4.Model.LICENSE - This means you are not permitted to use the Sugoi model for any expressly commercial purpose. - To use the model commercially, contact JParaCrawl/NTT and ask for a license suitable for commercial use. - To the best of my knowledge, the above software are the only components that are closed source and/or have significant limitations on their licenses. Everything else here if FOSS. - Since there are a lot of libraries used in the repackage as dependencies of other libraries, similar to how PyTorch uses CUDA, only the top-level components are listed here for brevity. - The repackaging scripts themseves are MIT. https://mit-license.org - Python's license. https://docs.python.org/3/license.html - git is GPL + LGPL. https://github.com/git-for-windows/git/blob/main/LGPL-2.1 - py3translationServer is AGPLv3. https://github.com/gdiaz384/py3translationServer/blob/main/LICENSE - For all the sub-libraries, type: - app\Python310\python.exe -m pip list - Search for the software library on https://PyPi.org to see their license. - Their source code is available on their project pages and also at apps\Python310\Libs\site-packages Edited March 5 by Entai2965 formatting, updated server links Quote Link to comment Share on other sites More sharing options...
Entai2965 Posted February 13 Author Share Posted February 13 (edited) Reserved. Edited February 14 by Entai384 Reserved means I was using this post to test the horrible formatting buttons. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.