Sample Assignment Project Ideas

AIM Sample Project Ideas

These are sample project ideas to get you started. You do not have to pick one of them. If you have your own idea, bring it, I would rather you worked on something you actually care about. Either way, discuss your choice with your lecturer before you commit to it. The same goes for datasets and for compute: the sets listed under each idea are suggestions, you are free to use another or generate your own, but talk to me first, and agree at the same time whether you are training from scratch, fine-tuning an existing model, or evaluating a pretrained one.

Contents

What you must submit

This is the same for every project on this list and is outlined fully in the assignment brief.

Scope and evaluation

For all the projects you should consider the following:-

Scope. You are not expected to train anything from scratch on a million images. Fine-tune a pretrained model, or train a small model on a subset, and spend the time you save on evaluating it properly.

A model that trains in an hour and is thoroughly tested is a better project than one that trains for three days and produces a single number.

Check the size of a dataset and the hardware you actually have before you commit to either; if the set is 200GB and you have a laptop, take a subset and say in your notebook that you did and why.

Most of the ideas below give you a narrow version to start with and something harder to extend to so get the narrow version working first as a proof of concept.

Evaluation. Every idea below has a suggested metric. Report it on an unseen test set, and report it against a baseline: a classical (non-learned) method, an off-the-shelf pretrained model, or a deliberately simpler version of your own.

Then show where it fails. Failure cases with an honest explanation get you more marks than a good average score with nothing behind it. “It looks about right” is not a result.

Splitting your data. Keep related samples together. Frames from the same video, clips from the same recording, variations of the same rendered scene and takes from the same performer belong on one side of the split or the other, never both.

Get this wrong and your test score is measuring memorisation, which is the single most common way one of these projects produces a number that means nothing.

Datasets. Check the licence before you download. A few of the sets linked here need a signed agreement that can take a couple of weeks to come back, which is fine if you start now.

Computer vision and 3D

Pose Estimation

Pose Estimation is the task of using computer vision techniques to predict and track the location of key features of a person or object in an image or video. Your system must take in an image or a sequence of video frames and calculate the estimated position of a human body.

Datasets: MPII Human Pose, VGG Human Pose Estimation, COCO Keypoints.

Reading: Zheng, Ce, et al. Deep Learning-Based Human Pose Estimation: A Survey. arXiv, 23 Jan. 2022, https://doi.org/10.48550/arXiv.2012.13392. Barla, Nilesh. “Human Pose Estimation: Deep Learning Approach”.

Evaluate with PCK or OKS, and look at temporal jitter if you are running on video.

Neural Denoiser

A neural denoiser is a class of machine learning algorithm which commonly uses supervised learning to reduce noise in a given image or video. In the visual effects industry it is used to reconstruct image sequences rendered with traditional light transport techniques at low sample counts. If you go the render route, feed the network the auxiliary buffers as well — albedo, normal, depth — because that is what makes the production denoisers work.

Datasets: Smartphone Image Denoising Dataset (SIDD) for photographic noise, or render your own paired low-sample and high-sample images in Blender or Houdini.

Reading: Tian, Chunwei, et al. Deep Learning for Image Denoising: A Survey. arXiv, 11 Oct. 2018, https://doi.org/10.48550/arXiv.1810.05052. Bako, Steve, et al. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings, SIGGRAPH 2017.

Evaluate with PSNR, SSIM and LPIPS, and show a temporal result — a denoiser that flickers between frames is no use.

AI Upscaling / Super Resolution

AI upscaling is the process of converting lower resolution media to a higher resolution. In a supervised setting, neural approaches train a model to predict a high-resolution image or sequence of images from a low-resolution input.

Datasets: DIV2K, Urban100: 4x upscaling, REDS for video.

Reading: Wang, Zhihao, et al. Deep Learning for Image Super-Resolution: A Survey. arXiv, 7 Feb. 2020, https://doi.org/10.48550/arXiv.1902.06068.

Evaluate with PSNR, SSIM and LPIPS, against bicubic upsampling as the baseline. Be aware that the perceptually best result often scores worse on PSNR than a blurry one, and discuss that rather than hiding it.

Object Segmentation

Object segmentation is the process of applying a label to every pixel in an image — for example isolating a subject in a photo and blurring the background to fake depth of field. Do not confuse this with object detection.

Datasets: COCO-Stuff, Cityscapes, Open Images 2019 instance segmentation, UCI Image Segmentation if you want something small to prototype against.

Reading: Minaee, Shervin, et al. Image Segmentation Using Deep Learning: A Survey. arXiv, 14 Nov. 2020, https://doi.org/10.48550/arXiv.2001.05566.

Evaluate with mean IoU, and report per-class as well as the average — the average hides the classes you are bad at.

Object Recognition

Object recognition is a class of computer vision techniques for identifying objects in images or videos. You are asked to train a classification model which takes an image or a series of images as input and correctly identifies the object.

Datasets: CIFAR-10, Handwriting Recognition, Food-101, Cancer Instance Segmentation and Classification if you want a domain where the classes are not everyday objects.

Reading: Liu, Li, et al. “Deep Learning for Generic Object Detection: A Survey.” IJCV, vol. 128, no. 2, 2020, pp. 261–318, https://doi.org/10.1007/s11263-019-01247-4. Zaidi, Syed Sahil Abbas, et al. A Survey of Modern Deep Learning Based Object Detection Models. arXiv, 12 May 2021, https://doi.org/10.48550/arXiv.2104.11892.

Evaluate with top-1 accuracy and a confusion matrix, and say something about the classes that get confused.

Monocular Depth Estimation

Predict depth from a single image, then use it for something — 2.5D parallax, depth of field, atmospheric fog, relighting. The downstream use is what turns this from a model download into a project. If real depth data is awkward to get hold of, generate your own: build simple scenes in Blender with a fixed camera and a handful of shapes, export the depth pass as ground truth, and add geometry and lighting variation once the fixed case works.

Datasets: NYU Depth V2, DIODE, or your own synthetic renders.

Reading: Ranftl, René, et al. Towards Robust Monocular Depth Estimation (MiDaS), TPAMI 2020.

Evaluate with AbsRel and δ<1.25 against a constant-depth baseline, remembering that monocular depth is only correct up to scale, so you need to align before you compare. Look hard at object boundaries, which is where these models smear.

Optical Flow Estimation

Estimate where every pixel moved between two consecutive frames. It underpins half the other ideas on this list — interpolation, matting, retiming — so it is worth understanding properly. Start with small synthetic translations you generated yourself, where you know the answer exactly, before going anywhere near real footage.

Datasets: MPI Sintel, which is ground-truth flow from an animated film, and FlyingChairs.

Reading: Teed, Zachary, and Jia Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow, ECCV 2020.

Evaluate with end-point error against the known motion, and compare against a classical method: Farnebäck in OpenCV is two lines and a fair opponent. Break the result down by fast motion, occlusion and untextured regions, because the average hides all three.

Image Inpainting

Reconstruct masked-out regions of an image. Paint-out and wire removal are the production versions of this, and you can build the training data yourself by masking intact images. Start with one image category and a controlled range of mask sizes.

Datasets: Places2, CelebA-HQ.

Reading: Suvorov, Roman, et al. Resolution-Robust Large Mask Inpainting with Fourier Convolutions (LaMa), WACV 2022.

Evaluate with reconstruction error inside the mask against a simple local fill such as OpenCV’s Telea method, and show close-ups of the boundary, as this is where a plausible-looking result usually gives itself away. Test on mask shapes you did not train on.

3D Reconstruction from Video

Capture a small object or a set on a phone, reconstruct it as a radiance field or a set of Gaussians, and render a new camera move through it. Good if you want a project that ends with a picture you would put in a reel.

Datasets: your own footage, plus Mip-NeRF 360 for a controlled comparison.

Reading: Kerbl, Bernhard, et al. 3D Gaussian Splatting for Real-Time Radiance Field Rendering, SIGGRAPH 2023.

Evaluate with PSNR, SSIM and LPIPS on held-out views, and report training time and frame rate as well. This is a field where speed is a really important factor.

Point Cloud and Mesh Segmentation

Label the parts of a scanned or modelled object. The geometry-first option if you would rather not spend the whole unit in image space.

Datasets: ShapeNet Part, ModelNet40.

Reading: Qi, Charles R., et al. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, NeurIPS 2017.

Evaluate with per-part IoU, and test whether your model survives rotation and resampling of the input.

Animation and VFX

Motion Capture Cleanup and Gap Filling

Take marker or joint data with dropouts and jitter and predict the missing frames. It is the denoiser idea applied to motion rather than pixels, and anyone who has cleaned a bad take will not need the motivation explained. Start with one skeleton and short gaps in walking cycles; long gaps and varied actions are the extension.

Datasets: CMU MoCap, AMASS, LaFAN1, or your own capture.

Reading: Harvey, Félix G., et al. Robust Motion In-Betweening, SIGGRAPH 2020.

Evaluate with L2 joint position error against the ground truth and against linear interpolation, plus foot-skate and a plot of joint velocity. A result that scores well and still slides the feet is not solved. Play the original, damaged and reconstructed motion side by side.

Motion Clip Classification

Classify short motion clips as walking, running, jumping and a handful of other actions. Less glamorous than generating motion, but it is a contained project with a clean evaluation, and preparing consistent skeletons and labels will teach you more about mocap data than you expect.

Datasets: CMU MoCap, NTU RGB+D if you want something larger.

Reading: Yan, Sijie, et al. Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition, AAAI 2018.

Evaluate with a confusion matrix against a classifier built on simple joint velocity features, and test on performers held out of training. The same actor walking in train and test is not a test.

Frame Interpolation and Retiming

Synthesise intermediate frames between two inputs. Very visual, easy to demo, and the places it breaks — occlusions, large motion, repeated texture — are the interesting part. You can build the training data from any footage you have by dropping alternate frames and keeping them as targets, but keep whole clips on one side of the split.

Datasets: Vimeo-90K, X4K1000FPS, or your own footage.

Reading: Reda, Fitsum, et al. FILM: Frame Interpolation for Large Motion, ECCV 2022.

Evaluate with PSNR and SSIM on the held-out middle frame against frame blending, and show a slow-motion result at 4x or 8x where the errors accumulate. Shot boundaries are worth testing deliberately. Most interpolators produce something horrible across a cut.

Video Matting and Rotoscoping

Predict a per-frame alpha matte without a green screen. Ties straight into compositing, and the temporal stability of the matte matters as much as any single frame.

Datasets: VideoMatte240K, Composition-1k.

Reading: Lin, Shanchuan, et al. Robust High-Resolution Video Matting with Temporal Guidance, WACV 2022.

Evaluate with SAD, MSE and gradient error on the alpha, and composite over a hard background to show the edges.

Audio-Driven Facial Animation

Speech in, blendshape weights or vertex offsets out. A good option if you are on the animation side and do not want a pure image project.

Datasets: VOCASET, MEAD.

Reading: Fan, Yingruo, et al. FaceFormer: Speech-Driven 3D Facial Animation with Transformers, CVPR 2022.

Evaluate with lip vertex error, and be honest that this one needs a subjective look as well. Play it back and watch it.

Fluid and Smoke Super-Resolution

Train on paired coarse and fine simulations so a cheap sim can be up-ressed. You generate your own data in Houdini or Blender, which sidesteps the dataset problem completely, though it does mean the sim time is now your problem.

Datasets: your own, generated as paired low-resolution and high-resolution runs of the same setup.

Reading: Xie, You, et al. tempoGAN: A Temporally Coherent, Volumetric GAN for Super-Resolution Fluid Flow, SIGGRAPH 2018. Chu, Mengyu, and Nils Thuerey. Data-Driven Synthesis of Smoke Flows with CNN-Based Feature Descriptors, SIGGRAPH 2017.

Evaluate with per-voxel error against the fine simulation, and render a sequence. Temporal coherence is the whole point.

Neural Skinning and Deformation

Learn corrective deformations on top of linear blend skinning from a set of simulated or sculpted poses. Small data, small model, runs on a laptop, and the result is a rig tool someone might actually use. Cloth is the more ambitious version of the same idea: keep the mesh topology fixed, start with static draping before you attempt animated cloth, and hold out some parameter combinations to test on.

Datasets: your own, exported from a rig or a cloth or flesh simulation.

Reading: Bailey, Stephen W., et al. Fast and Deep Deformation Approximations, SIGGRAPH 2018.

Evaluate with per-vertex error against the simulated ground truth, and report the speed-up over running the simulation. A deformer that is accurate and no faster than the sim has missed the point. Look at collisions and interpenetration, which the average vertex error will not show you.

Material Capture from a Single Photo

One flash photo of a surface, generate albedo, normal, roughness and metallic maps. Render the result back under new lighting to show it works. A smaller version, if the full SVBRDF problem looks like too much, is to predict just roughness and base colour from renders you generate yourself with fixed geometry, lighting and camera, then vary the lighting once that works.

Datasets: Deschaintre synthetic SVBRDF set, MatSynth, or your own renders.

Reading: Deschaintre, Valentin, et al. Single-Image SVBRDF Capture with a Rendering-Aware Deep Network, SIGGRAPH 2018.

Evaluate with parameter error and a rendering loss: compare renders of your predicted maps against renders of the ground truth under lighting the model never saw. Hold out some material combinations entirely.

Texture Defect Detection

Find the scratches, missing patches and seams in a texture, and say where they are rather than just that they exist. Asset QA is a real and tedious job, and this is the version of anomaly detection where you train on clean examples only, because you cannot enumerate everything that might go wrong.

Datasets: MVTec AD, which gives you clean training images and annotated defects to test on. Start with a single category.

Reading: Bergmann, Paul, et al. MVTec AD — A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection, CVPR 2019.

Evaluate with image-level detection AUC and the overlap between your predicted region and the labelled ones against a simple patch-distance baseline. Check what normal variation in the texture does to your false positive rate. That is what decides whether anyone would use it.

Line Art Colourisation and 2D Inbetweening

Either colourise cel drawings from a reference frame, or generate inbetweens from two keys. Both are real problems in 2D production and both are much harder than the 3D equivalents because there is no motion field to lean on.

Datasets: ATD-12K, Danbooru2021.

Reading: Siyao, Li, et al. Deep Animation Video Interpolation in the Wild, CVPR 2021.

Evaluate with PSNR and SSIM, but lean on side-by-side comparison as well.

Audio and sound

Dialogue Cleanup for Production Sound

Take noisy, reverberant location recordings and predict a clean dialogue track. Build your pairs by mixing clean speech with noise at known levels, and keep speakers and noise recordings apart across the splits.

Datasets: VoiceBank-DEMAND, DNS Challenge.

Reading: Défossez, Alexandre, et al. Real Time Speech Enhancement in the Waveform Domain, Interspeech 2020.

Evaluate with PESQ, STOI and SI-SDR against spectral subtraction as the baseline, and include audio you can play in the notebook, then judge it on intelligibility, residual noise and how much the speech itself has been damaged.

Stem Separation

Split a mix into its parts. Two useful framings: music into vocals, drums, bass and other, or a finished film mix back into dialogue, music and effects, which is what an M&E deliverable actually needs. Vocals against everything else is the version to start with.

Datasets: MUSDB18-HQ for music, Divide and Remaster for the film mix version.

Reading: Rouard, Simon, et al. Hybrid Transformers for Music Source Separation, ICASSP 2023.

Evaluate with SDR or SI-SDR per stem, and listen for what leaks between the outputs.

Subtitling and Caption Alignment

Transcribe, then do the part that is actually hard: split the text into readable subtitle lines, time them to the speech, and respect reading speed and line length limits.

Running Whisper is not the project; the segmentation and the timing are. A narrower version, if you want a contained project, is timing correction alone. Assume the transcript is correct, align it to the speech, and output the SRT.

Datasets: AMI Meeting Corpus, VoxPopuli.

Reading: Radford, Alec, et al. Robust Speech Recognition via Large-Scale Weak Supervision, ICML 2023.

Evaluate with WER for the transcript, and separately measure start and end timing error against hand-aligned subtitles and how many of your subtitles break the reading-speed guidelines. Show what happens on long pauses and on rapid speech.

Speaker Diarisation

Work out who spoke when across hours of unlogged audio, so an editor can jump straight to a contributor. Documentary and podcast rushes are the obvious use.

Datasets: VoxConverse, AMI.

Reading: Bredin, Hervé, et al. pyannote.audio: Neural Building Blocks for Speaker Diarization, ICASSP 2020.

Evaluate with diarisation error rate, broken down into missed speech, false alarm and confusion so you can see which part is failing.

Neural Foley

Generate or retrieve a sound effect that matches and syncs to a short video clip. Retrieval from a sound library is the tractable version and makes a better demo than generation, so start there.

Datasets: Greatest Hits, VGGSound.

Reading: Luo, Simian, et al. Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models, NeurIPS 2023.

Evaluate with retrieval accuracy and onset synchronisation error, and put the result on the video so it can be judged.

Search a sound library by description — “heavy footsteps on gravel” — rather than by filename. Start from pretrained audio and text embeddings and see whether adapting them to your library helps; the honest answer might be that it does not, and finding that out properly is a result.

Datasets: your own library, plus Clotho or AudioCaps for text descriptions paired with audio. VGGSound is useful for related classification experiments, but a retrieval project also needs written queries and relevance labels, and writing those is part of the work.

Reading: Elizalde, Benjamin, et al. CLAP: Learning Audio Concepts from Natural Language Supervision, ICASSP 2023.

Evaluate against filename search on a labelled query set, reporting how often a usable effect appears in the first few results rather than only Recall@1. In practice a sound editor auditions the top handful.

Audio and Video Synchronisation

Detect the timing offset between sound and picture and correct it. Make your own data by recording synchronised clips with a visible event such as a clap or an impact, then shifting the audio by a known amount.

Datasets: your own recordings, plus LRS2 if you want to attempt the speech version.

Reading: Chung, Joon Son, and Andrew Zisserman. Out of Time: Automated Lip Sync in the Wild, ACCV 2016.

Evaluate with the error between your predicted offset and the one you introduced, against a baseline that simply matches peaks in the audio envelope to peaks in visual motion. Speech is the harder extension, and it is much harder than claps.

Music Tagging for a Production Library

Predict mood, genre and instrumentation so a library track can be found by description. This is what production music libraries pay people to do by hand.

Datasets: MTG-Jamendo, FMA. Avoid GTZAN — it has duplicates, mislabels and repeated artists, all documented in Sturm’s The GTZAN dataset: its contents, its faults.

Reading: Won, Minz, et al. Evaluation of CNN-Based Automatic Music Tagging Models, SMC 2020.

Evaluate with ROC-AUC and PR-AUC per tag, and split by artist so the same act cannot appear in train and test.

Editing, broadcast and delivery

Shot Detection and Rushes Logging

Cut a long file into shots, classify each one such as wide, mid, close, who is in it etc, and output something an editor can load. The output format is part of the project. Start with hard cuts on a small set of clips you have annotated yourself, and state the timing tolerance you are scoring against.

Datasets: ClipShots, MovieNet.

Reading: Souček, Tomáš, and Jakub Lokoč. TransNet V2: An Effective Deep Network Architecture for Fast Shot Transition Detection, 2020.

Evaluate with precision, recall and F1 on shot boundaries against frame differencing, reported separately for hard cuts and gradual transitions, because the second is where everything falls over. Look at what camera movement and flash frames do to your false positives.

Video Summarisation and Highlight Detection

Pick the moments worth keeping from a long recording. Sports and lecture capture are the obvious cases and both have data.

Datasets: TVSum, SumMe.

Reading: Apostolidis, Evlampios, et al. Video Summarization Using Deep Neural Networks: A Survey, Proceedings of the IEEE, 2021.

Evaluate with F1 against the human summaries, and read the survey’s section on why that metric is contested before you rely on it.

Automatic Reframing

Take a 16:9 master and produce a 9:16 or 1:1 version that keeps the subject in frame, with a camera path that does not jitter. The temporal smoothing is where the marks are. Start with one person in shot; multiple subjects, and deciding which one to follow when the answer changes mid-shot, is the extension.

Datasets: DHF1K for saliency, YouTube-VOS for tracking.

Reading: Google’s AutoFlip is the non-learned baseline to beat.

Evaluate with how much of the ground-truth subject stays inside the crop, plus a smoothness measure on the crop path, and show the result next to a fixed centre crop.

Colour Grading Transfer

Learn a colourist’s look from before-and-after pairs and predict a 3D LUT for a new shot. Predicting a LUT rather than pixels keeps it fast and gives you a .cube file you can load in Resolve (or use OpenColorIO), which makes a much better demo. A simpler framing of the same problem is shot-to-shot matching for continuity: make your pairs by applying known exposure and white balance changes to your own footage and learn to undo them.

Datasets: MIT-Adobe FiveK, or your own graded footage.

Reading: Zeng, Hui, et al. Learning Image-Adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-Time, TPAMI 2020.

Evaluate with ΔE in CIELAB against the graded reference and against histogram matching, and show a shot graded by your LUT next to the colourist’s version. Check skin tones, clipped highlights and whether the result flickers across frames.

Archive Footage Restoration

Find and repair dust, scratches and damaged frames in old footage. Build paired training data by adding controlled damage to clean footage, one type of damage at a time.

Datasets: your own damaged pairs built on top of clean footage such as DAVIS, plus real archive material to test on.

Reading: Wan, Ziyu, et al. Bringing Old Films Back to Life, CVPR 2022.

Evaluate with reconstruction error in the damaged regions against a simple spatial or temporal median filter, and check the repairs stay consistent from frame to frame. Test on genuinely damaged footage separately from your synthetic set and be honest about the gap between the two. Synthetic scratches are always tidier than real ones.

Compression Artefacts and Video Quality

Two related projects, pick one. Either clean up the banding, blocking and mosquito noise on a low-bitrate stream, or predict how badly a viewer would rate a clip. The second is a regression against human opinion scores, which is a genuine change of pace from everything else on this list.

Datasets: LIVE-VQC, LSVQ, YouTube-UGC.

Reading: Li, Dingquan, et al. Quality Assessment of In-the-Wild Videos, ACM Multimedia 2019.

Evaluate with PSNR and SSIM for restoration, or SROCC and PLCC against the human scores for assessment.

Multimodal and text

Searchable Rushes

Caption or embed every shot in an archive so it can be found by typing “wide shot, harbour, overcast”. Cheap to build on CLIP embeddings, and the interesting work is in the evaluation: does it retrieve what an editor actually asked for?

Datasets: MSR-VTT, ActivityNet Captions.

Reading: Luo, Huaishao, et al. CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval, Neurocomputing 2022.

Evaluate with Recall@1, @5, @10 and median rank, and write your own set of realistic queries to test it on as well as the benchmark ones.

Alt-Text Generation

Generate image descriptions that are genuinely useful to a screen reader user, which is a different target from a generic caption: it needs the detail the user is missing, not a list of everything in frame.

Datasets: VizWiz-Captions, Conceptual Captions.

Reading: Gurari, Danna, et al. Captioning Images Taken by People Who Are Blind, ECCV 2020.

Evaluate with CIDEr as a starting point, then argue about why it is the wrong metric here and propose something better. That argument is the project.

Headline and Summary Generation

Abstractive summarisation with a hard length constraint. Worth pairing with a hallucination check, since invented facts are the whole story with summarisation in a newsroom.

Datasets: XSum, CNN/DailyMail.

Reading: Maynez, Joshua, et al. On Faithfulness and Factuality in Abstractive Summarization, ACL 2020.

Evaluate with ROUGE and BERTScore, plus a faithfulness check: count how many of your summaries contain something the source does not support.

Synthetic Media Detection

Decide whether a face, a voice or a whole clip was generated. Comes with an ethics discussion for free, and with a generalisation problem: a detector trained on one generator usually collapses on the next one.

Datasets: FaceForensics++, ASVspoof for audio. FaceForensics++ needs a signed agreement, so request it early.

Reading: Rössler, Andreas, et al. FaceForensics++: Learning to Detect Manipulated Facial Images, ICCV 2019.

Evaluate with AUC and equal error rate, and specifically test cross-generator: train on one manipulation method, test on another, and report how far it drops.

Embed a studio asset library, including models, textures, HDRIs and cached sims, then search it by text or by example image. The one idea here that is about the pipeline rather than the picture, and a realistic thing to be asked to build in a studio.

Datasets: your own asset library, or a public one such as Poly Haven or Objaverse.

Reading: Radford, Alec, et al. Learning Transferable Visual Models From Natural Language Supervision, ICML 2021.

Evaluate with Recall@k on a query set you write yourself, and time the search. A library search that takes ten seconds will not get used.

Co-Speech Gesture Generation

Generate body gesture from speech audio. Good project, but be warned that the evaluation is genuinely unsolved: there is no single correct gesture for a given sentence, so the automatic metrics are weak and you will need a small user study to say anything convincing.

Datasets: BEAT, TED Gesture.

Reading: Yoon, Youngwoo, et al. Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity, SIGGRAPH Asia 2020.

Evaluate with FGD and beat consistency, and plan the user study from the start rather than bolting it on at the end.

Other useful reading material

SE for Media ML AI Python PyTorch

Last updated on Jun 2, 2020

Related