1 Are you center punching the hole layout? 2 drilling in a drill press with correct pressure? 3 is the drill sharp and running at correct speed? 

In my experience if it isn't an annealing issue, it is one of those three things that are the cause of the issues in drilling.   

MP

Ditto to Matt's comments. One of the most difficult things to "unlearn" a person of, is keep them from trying to cram a drill bit through whatever they're drilling, as hard and as fast as humanly possible.   Good quality, sharp drill bits, and take your time.   

  I teach students what I call "tap drilling"..... put just enough pressure to start a small curl/chip, then let up and clear the chips from the bit.  Then repeat, repeat, repeat, and finally do all you can to overcome that urge/instinct to slam the drill bit through when you feel it about to go through the other side! 😆   It still amazes me that even folks who've used tools/drills their entire lives, try to go as hard and fast as possible when it come to drilling a hole.

Ditto x. 2 to Matt & Eds comments, especially the speed and pressure. The more pressure you put on a bit, the mor it will either flex or 'walk' out of alignment. What I will add is this, use a drill/cutting lubricant every time. I use and recommend Bo-Lube. Developed by Boeing Aircraft and will even help less than sharp bits cut better. Bo-Lube also helps the bit cut the size hole it is supposed to. 

Here is a foolish question from a proud fool: this is a center punch, right? I use it for marking things like how much material to use to make the handles, where too hot cut, and I was using it for hot punching(which is why its discolored), but now I use it for drifting as I made a smaller one for punching. That's the shape of the mark it makes. The blade is one I drifted the holes in just as something to compare. 

My modus operandi for measuring the holes is to use a carpenter/rafters square(i have had it for decades) to mark the handle, first measuring how wide it is, then using a welder pencil (or sharpie), then three lines using it as its a perfect 90-degree angle, then I use the shared punch to make a mark with it sitting on my avail. I do normally make 4 for a handle this size as i like to peen my pins instead of relying on just epoxy. I will do a furnace cool or vermiculite cool on the tang (i don't always bother to anneal the entire blade as I don't have a big enough bucket of vermiculite)

I use about the same amount of pressure I would for drilling through a piece of wood, and actually so rarely use this size bit as i bought it mainly for paracord wrapping...a nd was sick most of last year, so I still have fresh ones, but I will resharpen with a drill bit doctor...and check to ensure there is no loss of temper on the bits from operational use before I do if there is I grind it out before sharpening it. I am running my press at 1100RPM for this size bit (my chart does not have the list for this size, so I split the difference).  I recently started to take a moment to pull it up and spray it with some wd 40 to cool the bit down when drilling into steel. I do try to force it sometimes when I hit a hard spot.

yes,  that is a center punch.  not sure what size that is but the speed seems a little slow but pretty close. 3/16- 1500 RPM 1/4-1200rpm. 

Oil is not cutting fluid or coolant , you don't want to use oil for drilling or especially for tapping.  will if work .. sort of will drilling dry work, yes will actual cutting fluid work better, 100 X yes. 

I stopped slow annealing several years ago switching to a spheroid anneal (last cycle at 1250 for 30 min, in kiln, or just showing color in forge barely red) in steels with a lot of chromium or other carbide formers I will sometimes harden first then run my cycle this seems to help prevent/break up carbides that eat tooling. 

If i was laying out a tang for drilling  i would. 1 Dykem the surface. (layout fluid I like blue)2 use a dial caliper Scriber and 6" scale  to lay out hole placement. 3 center punch the hole locations. 4 select dill size and set drill press speed. 5 using a back up that is parallel to the table drill the holes.

When drilling I don't peck unless there is not a chip exiting the hole, with correct speeds and pressure, the drill should pull a chip out of the hole as long as the chip is not blue and is clearing the hole there is no reason to stop drilling.  blue indicates a dull drill to fast a rpm or wrong pressure, (to low or to high) pecking (in a and out of the hole) can lead to heat from not clearing chips or work hardening from repeated stops and starts. I use a consent pressure, backing off as the drill breaks through on the back side. This works well down to around 1/8 and up to 3x the drill with depth, below 1/8 and deeper than that I find I need cutting fluid and/or will need to clear the hole periodically.  ( i like tap magic or cool tool II) 

MP

Thanks. The bit size is 5/16ths. For now, I think I will keep using my method to make the lines just because I find it's currently what works for me for laying out the lines; I did however forget to mention I do use a caliper first to get the center. I will, however, change how I drill...and hunt for a better speed chart...what is a backup?

I do use actual thread cutting flued Do it best-Thred Cutting fluid, but I like to make a small divot before I commit to drilling, so the fluid has a place to stay instead of going all over the place...I will give tap Magic a try once I run out of this stuff.

Side Note: using WD 40 as a coolant is something I recently started doing between holes and more because I just have a spray can of the stuff around...and won't freeze on me. I will not use it anymore.

I did some forging today, so I did some testing between heats of my drill press with some mild steel scrap (which I always keep in a separate bin from my hardenable scrap, which is how I know its mild), and it was chipping out fine, I did get a little ribboning and am wondering if that is ok? if not, should I speed up or slow down my press? I did not bother lining it up perfectly as I just wanted to make sure the press was working right (I waited until I was forging because it's cold in my shop in winter as my forge is my heater).

Then I tried a piece of other steel and hit a hard spot it was a  blade made from what used to be a Ford truck leaf spring(when I use salvage, I only use Ford truck leaf springs i dought they would take time to reformulate as the do produce them on sight from ore). I think I needed to give the tang blade a little more soak time before letting it cool... and take some more time to line up the press I also realized the laser sight on my press is misaligned. it's got a computer-controlled VSD drive which i wanted...but it's not a high-end press...the lazer is likely why i was off by a quarter of an inch..not going to use that anymore. 😓 

ribbons are fine , as long as they are exiting the hole. 

from your comment's I think you are misunderstanding the point of center punching. The center punch mark is not to give a visual location,  it is to keep the bit from "walking" out of the location, the center of the drill bit should "seat" in to the center punch mark, relying on a laser pointer is not likely to work. 

in a stiff set up like a Mill you can spot drill with out a center punch mark, but even then when accuracy is important a hole is started with a center drill first or laid out and center punched first. 

MP 

Thank you for explaining that. The reason I brought up the laser pointer is I l thought that dot was aligned with the point of the bit. the press I learned to use back in high school did not have this kinda thing we were taught to use the bit to align it to the mark and check that it was lined up before we drilled into our material. I got lazy and thought the pointer's dot was where the point of the bit was to go.

Om what's a Stiff setup and how is it different from a drill press setup?

a stiff set up would be a Milling machine, Jig bore or lathe. 

MP

All good advice from Ed and Matt! Here's one more thing to consider. You stated you were using a 4 1/2" shank jogger bit. If you are just drilling through steel that is 1/4" I would suggest getting a good quality shorter shank bit. MSC direct has any bit you could ever imagine. 

Thank you I will give shorter bits a try if I still have the drifting issue, as its looking like me being lazy and using that laser. What's MSC Direct?... I tend to like to get my drill bits off amazon in bulk. They do have industrial machining stuff...and since I am a member of Amazon prime, I don't always have to pay for shipping and my budget right now is very limited since I don't want to sell my work until I have passed the Journeymen rank shop performance test.

Side note: I failed it last year due to not understanding Edge geometry as I should. the smith I did the test with (Howard Clark) explained what was wrong. It was on the rope, not the 2x4, if anyone is curious. I am getting another knife ready for attempt 2.