How do you feel about usage of AI in VN development?

[grayest]

AI is more or less humanity's next big creation; another thing that will change our lives forever, and is already has impact on something an niche as VN development, in spite of AI still arguably being in its infancy.

Recently, there's been lots of debate about AI in the creation of art, music, even voice acting, and nearly any kind of creative medium you can think of. Multiple VN developers have either completely used AI to make assets, or used AI to assist in doing so. (I won't mention who, or call anyone out)

Currently, I am writing a think-piece for Fuwa about this recent phenomenon. Not particularly to take side on one issue or the other (although I personally learn towards anti-ai for creative endeavors) I want to make this think-piece to spark conversation about AI in VNdev (but also in general)

I will be making a Stawpoll looking for responses which you can vote here. The responses will be used in the article. I will also be using replies in this post for the article/think-piece that I think have well-articulated points, or something worth mentioning. Therefore, keep that in mind when posting.

Thanks! - Grayest

[Tulip]

In the short-term: good only for things which the AI is actually good at.

The current state of the art in AI-generated visual art is already pretty good. It can pretty easily be used to generate high-quality backgrounds; it can potentially, depending on the broader artistic considerations at play, be used for CGs as well; and I suspect that there exist ways to get it to do even sprites to a reasonably-okay degree of quality, although I myself haven't yet learned the techniques.

But AI-generated writing, currently, is pretty consistently unimpressive. While authors have found various clever ways of incorporating AI into their workflows in small ways—as a source of sentence-continuations to get unstuck (followed by totally overwriting the likely-not-very-good sentences the AI wrote), or as a source of character-names, or suchlike—we're still a long ways off from having any text-AI tools that can generate actual high-quality storytelling. Perhaps we'll be there some day, but for now, I don't expect any VN that makes extensive use of AI-written text to end up particularly well-written.

Voice acting is somewhere in between the two, going by the AI voices I've encountered thus far. Noticeably worse than human voice actors, for now, with less grasp of timing and of emotional nuance; but still potentially better-than-nothing, for some readers. (And, of course, in any modern VN engine, readers for whom AI-generated voices are grating can always just turn off the voice audio-track, so there's not as much harm done by voices which only some readers like as by text or art which only some readers like.)

I'm sufficiently uninformed on the current state of AI-generated music that I lack opinions here. I'd be interested in learning more, if anyone in this thread happens to know things about the current state-of-the-art tools available to the public AI-music-wise!

(And probably eventually we'll get some real options emerging even for AI animation, but that seems a ways off still. Although 'a ways off' is, perhaps, less significant than it sounds, with how quickly AI-development has been happening lately.)

Overall, then, I think the state of my opinions on use-of-AI-in-visual-novels, relative to the current state of the tech, is that it can be used for some parts of a VN much better than for other parts. The only area where I currently expect AI to actually compete quality-wise with non-AI creative tooling, right now, is visual art; for everything else, I expect AI to be—for now—purely the budget option, usable by creators who would otherwise be unable to pull together anything at all (e.g. for voice acting, which is otherwise expensive-and-logistically-challenging to set up) but not likely to produce VNs anywhere near as good as those created by more traditional techniques. For art in specific, I do expect VN-artists to be able to make pretty good use of AI in place of older tools in many cases, and expect the art-quality of low-budget VNs to be raised thereby.

(Certain social spheres are big on claiming that there are ethical issues of some sort around use of AI art. The more-sensible ones on the basis of its causing technological unemployment for artists, increasing the abundance of quality art for the common person and thus reducing the market for specialized art-creators; the less-sensible ones on the basis of very implausible copyright-related claims, which if taken seriously would imply that a human drawing art inspired by the work of another artist is inherently in violation of that artist's copyright even if the art is different in all the particulars. I grant that the technological unemployment is going to happen, and that it will be a downside of the spread of AI-art; but I have a very hard time seeing "this product becomes more abundant, to the detriment of those who were profiting off of its scarcity" as being a bad thing overall, even if it's bad for a subset of people. The world is richer for the existence of cheap mass-produced clothing, despite the Luddites' objections and efforts to slow the adoption of the machinery that enabled that mass-production and despite the losses that many of the aforementioned Luddites genuinely suffered as a result of that mass-production; I would be very surprised if it ended up going any differently with art, or with whatever other media cheaply-accessible AI next moves into being seriously-competitive-with-human-creators-in.)

In the long-term: AI tooling is likely to become competitive with traditional tooling, while more accessible to the general public, in increasingly-many media. Even if visual art is, for now, the only medium where it can really keep up, I expect that to change eventually, letting increasingly many of the multimedia elements of VN-creation be done via AI.

As this process progresses, it will become increasingly practical for small teams or even individual hobbyists to create VNs of a scale and quality that was previously only reachable by large well-paid teams. This increased ease-of-production will, in turn, increase the accessibility of such VNs to readers: there will be more good stuff for us to read, much as the self-publishing revolution has led to greater abundance of high-quality traditional prose novels. (Along with lots of lower-quality ones; but it's not hard to skip past and not-read those, benefiting from the upside without substantial downside. I expect similar things to happen in the VN-sphere: the importance of reviews, and other such curation tools, will go substantially upward, since they'll become a core means by which people can find the good stuff amid the flood of less-good stuff.)

On the whole, I expect this to be a pretty straightforwardly good thing. The more VNs are made, the more good VNs will be made, and even if it takes some effort to sift the good ones out from the bad, it will lead ultimately to a richer and more-fulfilling range of options for VN-readers.

(Meanwhile, from the creators' side, I expect it to be an era of unprecedented flourishing for the small non-financially-motivated creators who, through their AI-use, can create VNs of vastly-larger scale and quality than would otherwise have been accessible to them. But, meanwhile, it will likely be an era of substantially harsher competition for the more-financially-motivated creators. Much as in the visual-art case, I expect this to be pretty straightforwardly to the benefit of the world overall, but with costs to those who are particularly benefiting from the current shape of the market.)

Edited August 19, 2023 by Tulip

[Clephas]

I've seen the AI-generated games that are popping up on DLsite, and I even played a few.  My first thought after doing so was... boredom.  They look beautiful and technically hit the fetish points of those trying to play them, but they somehow manage to be completely bland and soulless nonetheless. 

[Mattyd]

OmOneko is being made with ZERO Ai.

 

Although I can see a lot of #gamedev's using AI in the future to make games.

Artist: create a sprite and then train Ai to creat expressions and poses.

Code, etc. same.

 

Eventually those that don't use AI (us) will be like artisans that hand make furniture...

YMMV

[Hamadyne]

I've been working in the VN space for over a decade now as both an avid hobbyist and professional software engineer. There is an immense swath of incredible content out there, all made by gorgeous human hands. In recent years, developers like me have asked the question: "What benefits are there to AI tools?"

AI-generated content for visual novels have a lot of pros and cons, such as:

Pros:

• Easier for low or no-budget developers to create art assets
  • Some developers prefer doing things on their own, or lack the funds required to commission the amount of content required for a full-fledged VN. Sprites, backgrounds, and CGs are typically the bulk of the artwork required for a VN of any scope, and being able to quickly generate some of them saves money and time. Done well, you can create some impressive-looking backgrounds and CGs using pure prompting, overlaying a preexisting image, or a hybrid of both.
  • Similarly, developers can use AI generated content as a means to get placeholder assets in-game quickly, and can be used as part of a demo, game jam, or beta version until the real assets can be created.
• AI image upscaling allows for resizing of images
  • Probably the easiest and simplest usage of AI. Very useful if you need to scale an image to a different size for zooms or pans.

Cons:

• AI-generated writing is typically of poor quality
  • There's no real comparison -- human-led writing is still leagues better than what current LLMs can output, and avid VN readers will be able to quickly sniff out an AI-generated eroge route.
• AI-generated music and voice acting is also of poor quality
  • Once again, real human creativity shines through here. AI-generated voices are often flat and monotone, and music is similarly mechanical in both tone and sound.
• AI models often contain copyrighted/unauthorized work
  • The ethics involved here make it difficult to really justify AI-generated content for commercial works. Most models available now often have art taken from social media or from platforms like Patreon, Pixiv, or Fantia without the artist's permission.

Other questions that developers have talked about:

Will AI tools cause some artists to get less work/be out of a job?

Yes, it's entirely possible that some artists will find it hard to compete with more advanced iterations of AI art models. Whether due to quality or quantity, some artists have become disheartened by the proliferation of AI as a means to an end. There are solutions to this (better self-advertising, seeking out paid work on job sites, etc.) but that is on the artist to learn.

Will AI tools increase the amount of visual novels released?

Most certainly. However, that doesn't mean that they'll be at Tsukihime or Stein's;Gate levels of quality. Truly stand-out titles work because of good marketing, a good response from the consumer base, and most importantly, good content. 100 AI-generated visual novels pale in comparison to one really well-made one. Do note that AI tools will most certainly ease the creation process of visual novels, as mentioned by @Tulip in their great post above. It's up to the team involved to make a truly great game.

 

In conclusion, I think there's some merit to using AI for parts of visual novel development. If you're on an extremely tight budget or prefer to do things yourself, then I think it's fine to use AI tools to get certain assets completed. Results can vary, but having a means to complete and share your story when you don't have the money otherwise is an admirable use of AI tools. It's good to use AI tools for:

• Prototyping your work/Creating assets on a budget
• Creating portfolio pieces
• Game jams
• Student work

However, if you're considering a large-scale commercial project, please consider saving your money and hiring other artists to help out. Consumers will also appreciate transparency when it comes to AI content. Consider adding a disclaimer explaining your usage of AI tools for your projects.

[Clephas]

One place where AI tools will probably shine is in artwork.  To be blunt, the only thing that stands out in AI-made games so far is the visual shine.  If you just want visual perfection, nothing does it like an AI.  On the other hand, if you want visual brilliance, it will probably be reliant on actual artists until the first AIs gain self-awareness and human independence vanishes forever, lol.

Oh and AI-made games' writing is...terrifyingly bland and informative without showing any signs of creative flair whatsoever.  You'd think that with all the examples of top-quality writing out there, the base quality would be at least slightly better, but it hovers around just below mediocre.

[Dreamysyu]

I once asked ChatGPT to write me a short story. It was very bland. Honestly, it wasn't even a story in the usual sense of this world, it felt more like a bad retelling. However, when I read it, I suddenly realized that I had played several human-made VNs whose stories were even worse.

[Sirus]

General thoughts

If machine learning research keeps going at the current pace, AI will replace most jobs that don’t require robotics, I think this could come in less than a decade. Society will have to change and adapt, something like UBI (Universal Basic Income) will have to be put in place or things will be dire.

I think this will happen whether we want it or not. The question is whether this will stay locked behind big companies controlling everything like OpenAI, digging the gap between the classes even more, or the revolution will be open sourced and the power will be in the hand of everyone (StabilityAI, Meta, all the smaller open source projects…).

It is my belief that creators should start getting familiar with these tools starting today and include them in their workflow to boost their productivity, especially if they’re working in a commercial fashion, or they are at a higher risk of being replaced first by people with higher productivity due to these new tools.

I fully understand the passion people have for their arts, but if it’s your livelihood, capitalism will have no pity for you, and you will have to follow the market to hope survive.

One of the most positive points for me is that it will usher a new area of creation for indies who work, like me, solo, and don't have enough lifetimes to learn all the different skills nor a bottomless piggy bank. I think we'll see more indie games of higher quality popup instead of the nth pixel art, 8bit music that's too common.

On the legality

From all I’ve read, these models are currently legal in the U.S. and E.U. as of now. This might change in the future and might be cause for concerns, but hopefully things will be cleared up soon. The worst that could happen are draconian rules which would make it easy to comply for megacorporations and impossible for small indies/open source.

On the ethicality

The ethical debate has been mainly centered on the art side since it’s one of those that’s the most easily distinguishable, I imagine with voice & music models getting way better it will spark it up again. I personally don’t think it’s immoral to train on copyrighted assets of others AS LONG as you’re not purposefully trying to copy their style. Models such as stable diffusion use billions of images and so the weight of each artist becomes very diluted (but some bigger artists can still be recognized thanks to the amount of art they have available). No one should train on someone’s creation with the goal of avoiding paying them. I think an exception is copying the style of a large production to make non-commercial fan content (see the recent debate with the Scooby-Doo controversy).

 

I will now go over the different domain that applies the most to us as game creators.

Art models

This is clearly what sparked the whole debate in the recent months. In a year since the release of Stable Diffusion 1.0, models have made giant leaps (SDXL, Midjourney V5…) and have an arsenal of tools (ControlNet…) making them some cases almost indistinguishable from human creations, be it drawn art of photography. While they can deliver stunning artwork, I think they still very much struggle to make anything consistent, which is a problem especially for something like a VN. I think these tools are best used by an already competent artist, as they could draw a sketch, feed it to ControlNet which keeps the consistency of what’s drawn, inpaint to get some of the detail rights, and finish up the fixes in photoshop.

Can you make a full game with AI art without any artistic skills? In my opinion, no. Not something that will look good or have any kind of artistic direction anyway.

Code models

I will be biased here as I’m a developer by trade. Code models (and by extensions more fully fledged LLM like ChatGPT) have taken the development world by storm, claiming you don’t need a developer anymore or whatever. That’s simply false, but not totally. At this current stage, they’re extremely effective as speeding up the workflow of a seasoned developer (GitHub reports up to 30% which is about what I would have said myself using it daily https://github.blog/2022-07-14-research-how-github-copilot-helps-improve-developer-productivity/), which could mean, for example, that a company could need one less junior developer to go with a senior developer, but that remains to be seen.

The effectiveness is also pretty dependent on the language used, with something simpler like C# or JS working pretty good, and something like Rust struggling more. I use it myself as more of an autocomplete at a single line, which it excels at. It’s also amazing at repetitive code with slightly changes, which it’ll suggest instantly. What it’s really bad at, is following long list of instructions, or trying to implement algorithms that are not already seen hundreds of times on StackOverflow or GitHub, and this is where it stops being useful to a novice, and why I don’t think the programmer job is gonna disappear anytime soon (nocode platforms haven’t killed us after all, and WordPress might be a huge chunk of the internet but it doesn’t stop people from recruiting more developers than those that exist).

I think this video is an excellent example, a man with no programming experience trying to make a simple platformer using only AI. Sure, he kinda managed to get something, but it took him 8 hours (or 4, I don’t really remember), when it could be whipped up in under an hour by a professional programmer (or probably even less than 10min without the assets). https://youtu.be/IyKKhxYJ4U4

There’s also a lot of these videos on twitter et al, of people whipping up a whole website in a single prompt with ChatGPT. It might work for some simple ideas, but good luck growing it up further. If all you have is a static page with some text then I guess it’ll work, but the average person won’t even know how to host a website.

Text models

ChatGPT also created quite a lot of turmoil, for good and bad reasons. I’ll mainly focus on 3 parts that could be relevant to us: Marketing, Creative Writing & Translation.

For marketing, I think it can be excellent, especially if you don’t like writing overly formal stuff. It can be a bit dry, but I think it’s wonderful if you need to generate a lot of PR material as indies are often pretty bad at that. I think this usage will grow more and more.

For creative writing… Well, it’s not good. It will write the most boring and generic stories, and the small context makes it impossible to keep track of all the elements for longer period of times leading to all kind of inconsistencies. I think having ChatGPT write your whole game would clearly be a mistake. Now there can be some more local uses. I like to use it sometimes to suggest different ways of phrasing or reformulating a thing (and I pick & mix what I like the best, if anything). It probably can be used for some more generic descriptions (that would be outside the narrative, like some menus or something). Also brainstorming some ideas, splitting tasks into some smaller tasks (that I do sometimes, and it suggests some things I could have forgotten to do). But please, don’t use it for creative writing, no one will want to read it.

For translation, there’s some good and some bad. First off, the good, is that it’s generally better than DeepL and Google Translate. You can feed it some context, it has a better understanding of the world and what the specific word mean in this context, you can nudge it with some additional infos (especially the name if you’re translating to/from Chinese or Japanese). Now the bad… It’s better, but nowhere near perfect, will trip up a lot, get the context or the subject wrong (especially with Japanese where it’s implicit a lot of time), completely miss jokes or puns… The list is long. If you were planning to hire a TL for a cents per word then I guess that it can’t be worse, but otherwise, please do your international players a favor and hire a professional translator.

Music & sound models

Honestly not much to say here, I don’t follow those too much, but they seem to be making some progress. Sound models seem to be able to generate some common sounds quite well (barking, quacking…) albeit with some random noise sometimes, but then again there’s already giant sound banks that exists so I’m not sure how useful that will be. Music models seem to be making some good progress as only a few months ago they sounded like some melody straight out of hell, and now its’ almost listenable garbage. But they will keep making progress, so we’ll see.

Voice models

This one has caused some controversies recently, as people have claimed their voice was used for training. The TTS models, which have been a thing since forever but only recently lost most of it’s roboticness are becoming straight up amazing, and something out of ElevenLabs which seems to be the SOTA (State of the Art) right now are almost indistinguishable from a real human voice if you add a little background music. Though the emotions and range of voices are still narrower and it clearly won’t replace the capabilities of a real VA for a more complex job right now, it could be exciting for something like a RPG with hundreds or thousands of characters with their own unique voice.

There’s actually multiple parts all making progress at the same time, there’s the voice generation where you can generate new voices without any input, which could algorithmically offer an infinite amount of voices. Then there’s the clearly most controversial one, the voice cloning, where you can recreate pretty accurately the voice of someone else from an audio clip as short as a few seconds (but the more audio and the higher quality it is, the best the results will be). This can lead to funny clips of Trump and Biden talking to each other, but more nefarious use like fake news or cloning the voice of a VA without their consent. This last usage must be firmly condemned, and might even be illegal from my short researches (https://higgslaw.com/celebrities-sue-over-unauthorized-use-of-identity/, but I’m not a lawyer). The last part is voice morphing. You might have heard of it recently, with those videos where someone sings the song of another artist. This is a bit different from voice cloning, the cloning part still exists of course, but you’re adapting the voice to a preexisting audio which can be damn convincing. I don’t really see any usage in video games for that last part since you would already need a competent singer to do the initial singing which is arguably the hardest part.

I think these models are gonna keep getting perfected and will lead to a lot less demand for VAs (except at the top, those who can sell a product by their voice alone). I can easily imagine some VAs, instead of coming in the recording booth, licensing their voice for a yearly usage, or by number of words etc.

 

I hope I didn't miss anything important, and I'm open to discuss with anyone with an open mind 

[Darklord Rooke]

On 8/20/2023 at 3:11 AM, grayest said:

I will also be using replies in this post for the article/think-piece that I think have well-articulated points, or something worth mentioning. Therefore, keep that in mind when posting.

Thanks! - Grayest

Sorry for the necro of this topic, I don't really have much time to browse internet topics anymore unforch so I'm out of touch with quite literally everything.

Keeping in mind that you're looking for well articulated, thought provoking and generally enlightened responses for this topic, I will say I think the use of AI in creative fields (as someone who works in the creative fields) is the suckiest suck that ever did suck, it sucks, boy does it suck, oh wow does it ever suck, who the hell thought of this because they really do suck?!?!?!

I mean, we know that this eventually will help small creators put out works on par to large creators, but we all know that large companies, which has a fiduciary duty to their shareholders, which means it's their duty to look for ways to make their shareholders money, in their all consuming bid to not just make a modest profit but try and make all the profits, then increase their share of all the profit again next year, will of course look to cut staff and use AI wherever they can. Which oneups their tendency to cut staff and foist their work onto everyone remaining for no extra pay by a considerably wide margin. It's why the writers guild in Hollywood went on strike. It's incredibly scary and worrying for everyone. Good for the consumer sure, but it's likely to put a lot of people out of work. I can't be on AI's side in this.

[Khushi100]

Yes, AI voice cloning and voiceover softwares are literally everywhere. They are good when they are actually helpful and also bad in many ways too.