WEBVTT 00:00.000 --> 00:01.916 Welcome to AI for Visual Effects. 00:01.916 --> 00:03.000 I'm Doug Hogan. 00:03.000 --> 00:06.916 In this video, I want to show you a really useful workflow for generating 00:06.916 --> 00:10.958 cleanplates and removing objects using ComfyUI. 00:10.958 --> 00:14.500 This setup uses inpainting powered by WAN 2.1 00:14.500 --> 00:20.083 together with Sam automatic masking to rebuild missing background areas in just a few minutes. 00:20.083 --> 00:24.333 In visual effects work, we often have to remove things from a shot in compositing. 00:24.333 --> 00:29.541 Things like tracking markers, unwanted props, reference actors for CG characters, 00:29.541 --> 00:32.875 or even sometimes just distracting background elements. 00:32.875 --> 00:39.000 Traditionally, that means rotoscoping, paintwork, projections, and a fair amount of manual patching. 00:39.000 --> 00:42.291 With Comfy, we can generate a cleanplate automatically 00:42.291 --> 00:45.291 and use that as the starting point for a final composite. 00:45.291 --> 00:49.291 And once the workflow is set up, it's really easy to run on other shots as well. 00:49.291 --> 00:52.208 So let's quickly walk through how this workflow works. 00:52.208 --> 00:54.500 A segmentation model creates the mask. 00:54.500 --> 00:59.000 Then an inpainting diffusion model reconstructs the missing pixels, 00:59.000 --> 01:03.833 and finally that result is merged back into the original frame. 01:03.833 --> 01:07.333 Everything is modular, so you can swap models, masks 01:07.333 --> 01:09.250 or settings depending on the shot. 01:09.250 --> 01:12.500 The first step is figuring out what we want to remove. 01:12.500 --> 01:14.250 Here I'm using two different options. 01:14.250 --> 01:16.916 The first is Florence 2 from Microsoft, 01:16.916 --> 01:19.750 which lets you describe the object you want to segment. 01:19.750 --> 01:23.000 The second is Sam 2 from Facebook, which helps us generate 01:23.000 --> 01:25.375 masks from the image using a points editor. 01:25.375 --> 01:29.625 So for this workflow, you just load your plate, pick the segmentation option 01:29.625 --> 01:32.875 you want to use, and then choose the region you want to remove. 01:32.875 --> 01:34.333 Both models have their strengths 01:34.333 --> 01:37.291 depending on the shot, so play around an experiment. 01:37.291 --> 01:41.458 Either direction you choose, you'll get a mask that isolates the desired object. 01:41.458 --> 01:45.625 And compared to manual garbage matting or frame by frame rotoscoping, 01:45.625 --> 01:47.958 this can save a huge amount of time, 01:47.958 --> 01:51.375 but if you already have a mask from compositing software like Nuke 01:51.375 --> 01:56.208 or After Effects or another pipeline tool entirely, you can bring that in as well. 01:56.208 --> 02:00.583 Just use a load video node and import your MP4 video 02:00.583 --> 02:02.750 containing your black and white mask. 02:02.750 --> 02:05.875 So integrating this into an existing visual effects 02:05.875 --> 02:08.750 workflow is pretty straightforward, as you can see. 02:08.750 --> 02:13.208 MP4 tends to work best for these masks since Comfy currently prefers 02:13.208 --> 02:17.958 this video format over image sequences, image sequences can be used, 02:17.958 --> 02:20.250 but that usually requires additional custom nodes 02:20.250 --> 02:24.000 outside of the normal Comfy ecosystem that aren't covered in this video. 02:24.000 --> 02:28.416 But this is an area of rapid development, so keep an eye out. 02:28.416 --> 02:32.208 Once the mask is ready, the next step is generating a reference frame. 02:32.208 --> 02:36.666 This helps guide the inpainting model when it reconstructs the missing area. 02:36.666 --> 02:38.291 There are lots of ways to do this. 02:38.291 --> 02:42.708 You could even paint something quickly in Photoshop, but in this workflow, 02:42.708 --> 02:46.000 I'm using the QWEN Image Edit model. 02:46.333 --> 02:49.250 Other open source models like Flux can also work well, 02:49.250 --> 02:52.625 and there are closed source options like Nano Banana 02:52.625 --> 02:57.666 that produce really nice results too, so this is a great place to experiment. 02:57.791 --> 03:01.166 Once the reference image is created, the inpainting model analyzes 03:01.166 --> 03:04.750 the surrounding pixels and generates a clean background that fits the shot. 03:04.791 --> 03:08.083 Since the model is being guided by the original plate and the reference 03:08.083 --> 03:11.916 image, perspective and lighting usually stay really consistent. 03:11.958 --> 03:14.250 For many shots, this already produces 03:14.250 --> 03:18.291 a usable cleanplate, but if the result isn't perfect, though, 03:18.333 --> 03:23.000 you can simply run the workflow again and pick the best version. 03:23.000 --> 03:26.000 Oh, and quick tip: try not to let the mask 03:26.000 --> 03:29.000 perfectly match the silhouette of the object you're removing. 03:29.041 --> 03:32.208 That shape can sometimes actually influence the model. 03:32.208 --> 03:36.333 If that happens, just increase the expand value on the grow mask 03:36.333 --> 03:41.166 with blur node to give the model a bit more context and rerun the workflow. 03:41.291 --> 03:44.708 Typically, a larger mask actually gives you a better result. 03:44.791 --> 03:50.125 All right, now let's run the workflow. After a few minutes, here is our result. 03:50.166 --> 03:54.166 The object is removed and the background has been reconstructed. 03:54.208 --> 03:58.166 This cleanplate can now be used for additional visual effects work. 03:58.166 --> 03:59.541 Pretty cool, right? 03:59.541 --> 04:03.208 If the result needs refinement, you can tweak the mask 04:03.250 --> 04:07.291 using nodes like mask growth, blur or expansion. 04:07.333 --> 04:11.458 Like I mentioned earlier, making the mask slightly larger often 04:11.458 --> 04:16.250 gives the model more context and produces a better reconstruction. 04:16.250 --> 04:20.791 But feel free to run multiple generations and simply choose the version you like best. 04:20.791 --> 04:22.083 Experiment a little. 04:22.083 --> 04:25.125 As you can see, using ComfyUI for a cleanplate generation 04:25.125 --> 04:30.291 can dramatically speed up paint and prep work in a visual effects pipeline. 04:30.291 --> 04:35.375 And that means more time for the really fun, creative work that we all love doing.