Tutorial: Using MIDI to Control Animation in After Effects | Part 1

Here's how to control animation with a MIDI controller.

Ready to test the limits of what's possible in After Effects? In this lesson you're going to unleash some real creativity with using a piece of software for something that it wasn't really built for in the first place. There's going to be some plugins and expressions that you'll need for this lesson, so head on over to the resources tab to grab those.  

‍

{{lead-magnet}}

‍

-----------------------------------------------------------------------------------------------------------------------------------

Tutorial Full Transcript Below 👇:

Music (00:13):

[Toto - Africa]

Joey Korenman (00:31):

Yo Joey here at school of motion and welcome to day five of 30 days of after effects. Today, we're going to get experimental and talk about how to get MIDI information into after effects to create animation driven by that data. If you don't know what MIDI is, don't worry. I'll explain that too. We're going to be sort of in and out of after effects today, because we're going to have to use another program logic to be specific to actually create the media information. Now, hopefully this is really interesting and gives you some cool ideas about unique ways to use after effects. Don't forget to sign up for a free student account so you can grab the project files, example MIDI from this lesson, as well as assets from any other lesson on the site. All right, let's check this out. So let's start out this first video by doing a quick primer on what MIDI is, um, for those you, that don't have any experience with it.

Joey Korenman (01:22):

Um, and for those of you that do, um, you know, who knows, maybe you'll learn something. So this pro this, uh, app that I'm in right here is called logic. Um, and there's a lot of different, uh, programs out there that you can work, you know, with audio. And you got pro tools, logic, Cubase, sonar. I mean, there's a lot, um, if you have a Mac, it comes with garage band, which can do MIDI. Um, I just like to use logic. So first let me just kind of show you what MIDI is, right. If I hit play

Music (01:52):

[Alice DJ - Better Off Alone]

Joey Korenman (01:57):

It just sounds like bad eighties music. So, um, so what MIDI is, it's a way of storing musical information. Okay. So this song, and I just downloaded this off of a free MIDI site. Um, you know, it's got six instruments here and each instrument has its own track. And if I click on each one, you can see that there are notes assigned to each one. So, you know, when you go into a recording studio, typically you have microphones and you record the audio and once it's recorded, you know, you can edit it and do things to it. But typically that's what you got, whatever you played, that's what you get with MIDI. It doesn't work that way. So with this tracks selected, I'm just going to hit play. Um, and you can hear what it's doing, and you'll be able to see exactly what all these notes mean. So, you know, when you go to a recording studio and you play a piano in front of a microphone, um, you've recorded an actual audio file, but when you record MIDI, all you're recording is the information about when you hit each key on the keyboard, how hard you hit the key, um, and you can, and it records just a bunch of data. And what's great about that is then you can change it. So if I take these two notes, I can move them around, right.

Joey Korenman (03:15):

And I've made the song even worse than it already is. So, um, so that's how many works and you can see that it kind of works on a grid system, right. Um, and this is great if you're, you know, if you don't actually know how to play an instrument, you can still kind of make music, you know, really simply, um, you know, just sort of, let me get rid of scared of some notes here. You can just sort of, you know, program in whatever you want. Um, just by adding notes, you know, there's a note,

Joey Korenman (03:50):

Right? And so that's how many works now. Um, you know, if you have a, if you actually play piano, if you have a keyboard, you can really easily record your own MIDI information. Um, and, you know, as an animator, this was intriguing to me because, you know, especially when you're doing animations that have to be timed out to music, or you want things to sync up to a beat or something like that, it can be very tedious and there's some ways to do it in after effects that, you know, there's plugins that can kind of try to, you know, figure out when the kick drums being hit and when the snare drums being hit and, and kind of read the audio file. But if you could actually get this information into after effects, that would be super useful. Um, so let's, uh, so first thing I want to do is I want to show you guys, um, how I got some media information that I thought would be useful.

Joey Korenman (04:40):

Um, and so I, I play drums. So I use a, an interface, um, called an Octo pad. And I'll link to a little bit more information about that. If any of you guys play drums, you're curious about it. Um, so what, what I have here is I have a track, um, and I've got a plugin that I use when I, uh, when I actually play drums. I'm in logic, it's called superior drummer. It's this really cool plugin that works for a lot of, uh, programs. And it lets you, uh, use MIDI to control this, this very realistic, you know, drum set sound. Right. Right. So I sort of, uh, I have that set up and now I'm gonna, I'm going to walk over and I'm going to sit down on the Octo pad and I'm just going to hit record. Cool. So that shows you exactly sort of how many works, um, as I'm playing the drums, logic is just recording basically on each little track here. Exactly what I played and the different colors are just showing you different velocities. Right. So, Okay. So you can, uh, you know, you can see that the red hits are the strongest ones and the green are the weaker ones. And, uh, what's really great, especially if you're a drummer is you can select all these notes and you can fix them real quick. Um, and so now everything's going to be perfectly in time.

Joey Korenman (06:23):

Uh, the ending's not in time, but, um, what are you going to do? So anyway, so there's a, there's our MIDI information. Okay. And what I want to do is, uh, you know, I want to be able to use this in a way that's useful and aftereffects. Okay. So I'm going to, I'm going to rerecord something and I'm going to make something very, very simple. Then I'm going to show you how to get that into after effects. And I'm going to show you sort of the basics of how to start working with that information. Um, in part two of this tutorial, I'm going to get into some pretty crazy expressions, which will let you really, really control stuff. So let me hop back over to the kit and I'll just do something really, really simple. Cool. So that was a, that was much simpler and this is going to be easier to work with an after fact. So let's just make sure that it sounds the way we want.

Joey Korenman (07:31):

Great. Okay. And what I'm looking for here, um, I want a little bit more variation, uh, in the, in the dynamics of the playing, right. And you can see a lot of these hits are red, but you've got a few of them in there, um, that are different colors. So what I'm going to do is I'm just going to select some of these and I'm just going to change the velocity of them, which is going to make them, um, a little bit different. This will make it a little bit easier to see what's going on in after effects. Um, so All right, maybe I'll make these two little, a little softer to See this is what's great about MIDI, um, especially for a musician. You really can, you can just tweet the crap out of this stuff, right. Maybe I'll make these two little, a little softer. There we go. This one can be a little softer. Cool. And then we'll make these, make these a little bit softer too. So it's more of a buildup at the end. There we go. There we go.

Joey Korenman (08:45):

Cool. All right. So first thing we need to do is export an audio file of this. All right. And I'll show you how to do it in logic. Um, in garage band, it's going to be a slightly different, um, set up. Um, but all you need to do is, you know, use the internet and figure out how you export an audio file out of your, uh, digital audio workstation program. All right. So I'm gonna just select this region and I'm going to bounce this track and are going to call this, uh, let me make a new folder here and we'll call this demo and this is going to be my Tom's audio. Cool. All right. Now you guys probably saw that I, um, I trimmed this little layer. There was kinda some extra stuff at the beginning that I didn't need. Um, and so I trimmed it, so it was just the part of the audio that I needed.

Joey Korenman (09:37):

Okay. Um, and so if I just export this as is from logic, what I found is that it actually exports even the notes that I thought I erased. So what I need to first do is come down here. Um, and in my edit menu, just say, delete MIDI events outside the region boundaries. Um, and again, this is how logic works and I'm on the newest version of logic. But if you, um, you know, if you're using pro tools or something, you just need to look up exactly how to do that. All right. And then we're just going to go file export selection as MIDI file. Okay. Um, and actually this error message is what I was hoping wouldn't happen. It's saying there are some events before position 1, 1, 1, 1. So it's actually saying that, uh, there are many events that it's sensing that are happening before the beginning of the song.

Joey Korenman (10:28):

Um, and actually, if I zoom in here, I think I see what it's doing. You see how this first hit is actually a little bit before the beat. There we go. So I just moved it. Um, and let's see if I try to export this. Now, if it works, export selection is MIDI file. Here we go. And we're going to call this, um, demo. Tom's audio MIDI. There we go. Now we have a MIDI foot MIDI file. So now let's hop into after effects. And, uh, let's talk about how to actually get this information into after effects. Ha I lied. We're not an after effects. We're actually in a web browser. And why is that? Well, there's not a built in way in after effects to get MIDI information in. Um, you have to use a plugin and this is the only one I found.

Joey Korenman (11:21):

Um, and it works great. Um, and, uh, I've actually mentioned this site in another one of these 30 days of after effects tutorials. Um, this guy has made a bunch of sort of free plugins and scripts. And one of them is, uh, a Mitty importer. All right. And so you go to his website and you download it, it's a script. So you need to follow the instructions to actually install it in the right place. Um, and then once you install it, it shows up in after effects in your window menu right here. Oh, M underscore Mitty. And here it is. Okay. So first thing I need to do is import that Tom, Tom audio. All right. So let's go to the demo folder. Let's grab the Tom's audio. Let's throw it down there. Um, and let's just preview that real quick.

Joey Korenman (12:18):

Great. There it is, works perfectly. Um, right. And so like your typical, you know, workflow to sync things to this would be maybe open up the audio way form, which, uh, if you guys don't know this, you double tap L on an audio layer, it brings up the way form. Um, and maybe adding some markers, you know, wherever there's a hit. Um, but now that we have the MIDI, we have a much more powerful tool, so let's import the MIDI. So what we're going to do is, uh, make sure that our O M underscore MIDI script is open, and we're going to click on this little tiny icon there. And this is where we tell it, which MIDI file. So navigate to the folder, there's the MIDI file hit. Okay. Um, now this script, right? It's not like a professional thing that you buy. And so it's sometimes it's a little bit finicky.

Joey Korenman (13:05):

Okay. Um, so I found that what works best is to make sure you have the correct comp open and select some layer in there. Okay. It just sort of helps tell this script that this is the comp. I want you to put the MIDI information into, and when you hit apply, it should work. There we go. All right. Now here's what it did. It created a no. And I usually just immediately renamed that Knoll MIDI. And you can see that it's put a bunch of effects on here. Now, the effect that it put on there is actually my good buddy, the expression slider, and it's renamed them channels, zero channels, zero channel zero, channel nine, channel nine, channel nine. And if I click on this Knoll and I hit you, let's take a look at what we got. I've got a bunch of key frames in there. Okay. Now watch, I'm going to play the audio. And I want you to watch this play head.

Joey Korenman (14:00):

Isn't that cool. These key frames line up to the audio now they're, you know, we need to know how to interpret them. Okay. And here's, you know, here's one of the, the issues that I have found with this. Okay. And, and I'm not sure if there's a workaround and if someone figures this out, please tell me. But, you know, we work, um, in video, we work in frames. Okay. And you know, this comp for example, is 24 frames a second. But if you, if you listen to the Tom hits at the end, right, right. They're happening pretty quickly. And the, you know, if I look, if I go back to logic right. And zoom out, so we can really see all my hits, you can see that there is, you know, there's basically nine hits, right. If we come into after effects and we look at the, the very last little bit here, you know, there's how many, there's three, four or five.

Joey Korenman (15:00):

I can't tell that kind of looks like there might be two in there kind of snuck in. And what happens is the plugin actually tries to put the notes on individual frames for you. Um, and it, and sometimes it just gets kind of screwed up. You see how there's like two notes right next to each other. Um, so when you play quickly, when notes are very close together, this script, it doesn't do a great job of putting things exactly where they should be. Now, for most cases, it's close enough. And in the way that I used it in the demo, uh, with, you know, the most amazing song ever written Africa, um, I deliberately kept my playing pretty simple, knowing that this is how it's going to get used. Okay. Um, but here let's, let's just play this one more time.

Joey Korenman (15:51):

All right. And if you listen to the beginning, there's three hits right. In a row, and we're seeing here's, here's three key frames, but then here's six key frames. Okay. So, and then over here, we've got this channel channels, zero velocity that has nothing in it. Okay. So the first thing I like to do is clean up the media information a little bit, and this channel zero has nothing on it. So I'm going to delete all three of these channels, zeros that it's given me. So now all I have is channel nine now in MIDI, there's a lot of different things that can be measured. So obviously the timing when I actually hit the drum as measured, but also how hard I hit the drum that is measured, right. That's the velocity. And if I click on this and I go into my graph editor, you can actually see,

Joey Korenman (16:45):

Right. You can actually see that the velocity matches up perfectly to my audio. Now, the pitch, the pitch on this, isn't going to change because I'm playing one drum. If you had a, um, a piano instrument, though, um, then you would have this, you'd have a, you'd have the same three kinds of channels. Right. You'd have the, the duration. Um, and the duration of a drum hit is generally just kind of an instantaneous thing, right? So that's why when I click this, there's not a lot going on here. Um, the velocity is changing, but when you're playing a piano, you actually might hold a note for a second and then let go. And so this will actually have more information in it for a piano that does for a drum. Um, you'll also have lost the information, and then you'll also have pitch information for the piano or for any instrument that has notes.

Joey Korenman (17:37):

Um, this chart will move up and down and kind of tell you the pitch, uh, of the instrument, which you could do some cool stuff with too, but for now let's keep it kind of simple. The drum doesn't have a pitch. So I'm also going to delete that. So now, uh, you know, the way that this kind of works is whenever I hit a drum and it lands kind of on a frame, I get a key frame on the duration right now. Here's our first problem with the duration. You can see there's four key frames, there's actually five hits. Okay. And it's not, it's, it's sort of didn't register the first one or something. Right. But on velocity, it did register all of them. Okay. And what happens with velocity is it actually, it puts an extra key frame at the end. Okay. Um, and that extra key frame kind of levels it back out.

Joey Korenman (18:32):

Okay. You see that? Um, so you can, you can delete that or you can just ignore it, you know, you could go through and just sort of delete each of these key frames if you didn't want to use that. Um, but, um, but you know, until, until you actually start using the data, you're not really sure how it's gonna work. So, um, you know, I just, for this example, let's just leave this as is, and let's, let's start doing something. Okay. So here's a quick and easy thing. And, and, and, and just so you guys know you're going to have to use expressions to do this. There's no other way, um, to really make this work. And, you know, again, if you've watched any of my after effects, tutorials, you know, I love expressions, that's kinda my thing. And, um, and so, you know, I, I want you guys to kind of get more comfortable with them.

Joey Korenman (19:22):

So all I'm going to do is I'm going to make a little circle here, and I'm just going to tie the scale of the circle to this. Okay. So let's look at the velocity here. So the highest it gets is 127. Okay. So if I, and it's interesting. So if I go here, if I go forward one frame, you can see that the values 1 27. Okay. The reason that number is 1 27 is because that's the, um, that's just the scale that MIDI works at. Um, if we go back into logic and take a look at this, um, let me open up this little guy down here. I'm gonna change my I'm gonna change this graph to note velocity. And so now you can see with all of the notes that there's kind of a corresponding dip in this velocity. Uh, if I hold my mouse over this note, it says, velocity 1 27. I hold it over this note, velocity 80. Right?

Joey Korenman (20:20):

Right. So this is the information that's going back into after effects. So that's the maximum 1 27. And the minimum is zero. Right. If I'm not hitting anything, then the velocity is zero. So what I want to do is, um, I want to put an expression on the scale of this circle. Okay. So what I want the scale to do is to look at this velocity slider. And I want it to scale from 100% when I'm not hitting the drum, maybe up to 200% when I'm hitting it as hard as I can. And when I'm hitting it in between, I want it to be somewhere in the middle. So the way we do that is, uh, we first need to set up a variable just to make this easier to read. So the value I want this expression to look at is this slider channel nine velocity.

Joey Korenman (21:10):

So I'm just going to make a quick variable. Val equals valves, just short for value equals this. We always have to add a semi-colon at the end. Okay. And then I'm going to use the linear expression, linear expression, and after effects is awesome. Um, it lets you take one number that's changing over time and map it to a different number over time. Okay. Um, so it, you know, if you had a value, for example, the velocity of a Mitty note going from zero to 127, but I want to map the scale of this circle to go from 100 to 200. That's exactly what linear does. And the way it works is you type in linear in print the seeds, you have to give it five arguments. The first one is, what value am I looking at? Well, we're looking at that variable. We just made called Val.

Joey Korenman (21:59):

Then the next two, uh, arguments are the minimum and the maximum number that is expected to come out of this, this thing near measuring. Okay. So we know the minimum is zero and the maximum is 1 27. Then the next two numbers are, what are we going to map to these two numbers? Okay. So when this value is zero, what should the result be? Well, when, when I'm not hitting a drum, I want the scale to remain at 100. So I'm just gonna type in 100. And when I'm hitting the drum, as hard as I can, I want it to go up to 200 and that's it. Okay. Oh boy, we've got an error. Yes. Here we go. The error message was telling me that this expression won't work because, and I'm glad you guys saw that because you know, you see, even when you use expressions all the time, you still screw them up every time.

Joey Korenman (22:54):

So a scale is expecting an X and a Y number. And I am only returning one number here. So I'm actually going to say this S equals linear. So now I'm setting up one more variable S which is going to store the, the value that comes out of that linear expression. And now I can return instead of one number, I can return an X and Y all right. Um, and the way you do that is with opening bracket and then the first number S comma, the second number S close the brackets. Okay. So all I'm doing is I'm telling after effects, I want this X number and this Y number to be the same, both of the numbers we'll get we'll, we'll come from this expression here. Okay. Um, and if you are very unfamiliar with expressions, maybe you should watch the intro to expressions and after effects, video. All right. So now this expression should work. Okay. So now let's do a quick Ram preview. Let me set this to half. So it goes a little quicker. Great. Okay. Um, and let's make this number a little bit bigger too. Let's make this like 500 and then let's make this number 50 so we can get a lot more kind of variation out of it.

Joey Korenman (24:15):

So that's kind of it in a nutshell, um, how to get media information into after effects and how to begin using it. Um, I want to show you guys, one other thing I ran into while I was kind of preparing for this. Um, and I'm not exactly sure why, but it'd be, uh, might be useful for you guys to know about this. Um, I recorded a, a different piece of audio here. Um, let me grab it. It's this piece of audio, which is just, um, let me hit L L here. Right? So this is, uh, this is a snare drum kind of just solo piece here. Right.

Joey Korenman (24:53):

All right. And what's different about this? Um, you know, from the, the Tom thing I did is I'm playing a lot faster, right? Like, there's just a lot more notes to it. This is also longer. This is about 22 seconds here. And here's what happened when I imported the MIDI information. Okay. So, um, that MIDI information is here. Okay. Uh, snare two. Um, and then let me hit apply. It brings it in. And what I noticed was I got MIDI information that went way out past my, uh, my actual audio file. And I thought that was weird. And, um, so here, let me delete this channel that has nothing on it. And the pitch channel, which we don't need. And let's look at the velocity channel. Okay. Let's zoom in a little bit here

Joey Korenman (25:47):

And you notice right away that it doesn't seem to be matching up to the audio at all. And I thought that's really strange, and I didn't really understand why there was so many extra key frames here. Um, so what I did was I, um, I moved this layer over, uh, and I'm just gonna grab, I'm gonna basically click on this slider and then hold shift and click on this, just to select every key frame. And then while I'm holding option, I'm gonna grab the very last key frame and move it to the left. And what I'm doing is I'm scaling all of these key frames. And so what I want to do is try to line up this last key frame with this last snare hit right here. Okay. So I'm basically just going in, um, and if you go into the graph editor and you select both of these, so you go to the graph editor, tap this twice, then hold shift and tap and it'll select everything. And if you have this little button check the transform box, you can actually scale them and scale them in between frames. So you can really line that last hit up. So all I've done is I've taken my MIDI key frames and scaled them and hit, now let's, let's play this

Music (27:01):

[fast drumming]

Joey Korenman (27:08):

So you could see now it lines up perfectly, and I actually did check it and it lines up all the way through to the end.

Joey Korenman (27:18):

So if you have a ton of Mitty notes and it's a longer piece, just know you may have to scale the key frames. Okay. Um, and this is actually, this is pretty great. This shows you exactly why using MIDI information can be so useful because look, how many key frames there are and, and how much information is here. And if you had to hand key frame that it would just suck. So this is a way better way to do it. Um, you know, and then just out of curiosity, um, why don't we, um, why don't I copy this circle here and I'll just paste it into this and we can see exactly how it reacts, um, to this piece of audio. Okay. So I'm just gonna double tap you to bring up all, everything I've changed. I'm on my circle layer, and this will bring up this expression. And, um, you know, here's something that's kind of cool because I, I named my no MIDI and in the expression, that's actually what it was looking for. It was looking for a layer named Mitty. It just worked and I don't have to change anything. So let's, uh, let's do a quick Ram preview here and just kind of play a little bit of this.

Joey Korenman (28:34):

So you can see that it's working. Um, no, obviously it's very jerky. Um, and it's not super useful yet. I mean, if you just wanted something really kind of flickery and cool, and maybe you just wanted a layer to turn on and off instantly in time with music, you can do a very simple expression like this and get that to work. If you want to have things be triggered and trigger particles and things like that, then it's a little more tricky. So in the next tutorial, that's what we're going to get into. So hopefully this was a good kind of foundation, a good base of knowledge. So when we get into the more advanced stuff, it'll make a little bit more sense. So thank you guys as always, I'll see you next time. Thank you so much for watching and make sure you check out part two of this tutorial where we'll actually get really deep into expressions to make this stuff all work the way we want to. And if you learn something valuable from this video, please share it around. It really helps us spread the word. And when you do that, we really appreciate it. So also don't forget to sign up for a free student account to access the project files for the lesson you just watched, plus a whole bunch of other goodies. So thank you so much. I'll see you next time.

‍