I've given it more thought and come up with an answer.
O5R isn't a distinct system, and it's not 5e that's been houseruled into something unrecognizable. It's 5e mechanics lightly molded and paired with the DM style and decisions to create an old-school feel at the table.
So what does that actually look like? The first, most important differences are:
1) Ignore balance.
You should still at least check a creature's CR to see how dangerous it is, but don't treat it as the deciding factor in whether to use a monster or not.
2) Cut out all the standard D&D/WotC lore.
Don't include things just because The Lore says they exist, and don't hesitate to bring out the gonzo. Throw true weirdness into the mix for your setting, give it some wondrous and bizarre style.
(The corollary to this is you have to explain it to your players clearly. Present them with the important setting details so they can make meaningful choices based on the conditions in the world you've made. Do this before the game starts, at session 0 at the very least, to give them plenty of time to ask questions and understand.)
3) Don't let the PCs start bogged down by a lot of backstory.
Starting characters should be mostly formless. A basic personality and a handful of details like hometown, family business or previous profession, and a like, dislike, or goal is plenty. Let the rest emerge through play and the characters develop instead of coming to the table as an established entity with a specific build in mind.
(Do make sure your players have their characters take a background in the game mechanic sense though. That's a sizable chunk of their abilities. They can reflavor the backgrounds however they like.)
After you've done that, address the more mechanical aspects:
A) Good plans don't roll.
If players do something clever or come up with an unorthodox plan that might work, reward them. They've done a good job. Their plan works without needing to roll for it. At the very least it should give them a partial success, positioning them to succeed later or to have an advantage in their current situation.
B) Move the focus away from combat.
This ties into ignoring balance because combat is more dangerous now and usually the worst option in a situation. Give players ways to solve their problems beyond "sword." A good way to do it is to put the focus on exploration. Have the PCs go out into the unknown and face obstacles that they literally can't fight. Rockfalls, deep chasms or cliffs, weather, crumbling structures, etc. Present them with things they need to work through or find ways around instead of a band of goblins that'll only slow them down for a few minutes.
There will still be some combat, but you don't need to change any rules or tweak death save mechanics to make it more lethal. You just need to make enemies smart. Have them use tactics, teamwork, have goals to achieve beyond plain slaughter, use the environment to their advantage, and care about their own lives. Sometimes the PCs have to retreat to survive, sometimes it'll be the enemies who cut and run; but it'll only take a few encounters with competent, coordinated enemies to teach your players that combat's not the default reaction and should be avoided if possible.
C) Homebrew!
Throw your own stuff into the mix, but don't alter the player-facing mechanics. Instead of trying to lop off sections of the system or make radical changes to basic rules, have your homebrew material add to the existing game.
Pay attention to what the players are interested in and create spells and items that play on that. What you make doesn't have to be powerful, just interesting. For example, when my players fought and slew a bus-sized magically mutated scorpion they asked if they could harvest the chitin to make armor. Of course, they could! (I'm so proud of them.) They took some, dragged it back to town, and found someone to work it into gear. The finished armor had the same stats as a breastplate plus a little magic resistance. Not an exceptionally powerful magical item, but better than what they had and cool to them because of the origin.
Same goes for rules. Listen to what the players want to do and find a way to make it happen. The same group of players found several statues and wanted to try auctioning them off, hoping to get more money than they would with a direct sale. So I made auction rules that leave the exact amount of profit up to luck and keeps the party engaged in a mini game.
Whatever you decide to add, be sure it makes the game more fun and the world more wondrous instead of just harsher.
That's mostly it. Also for consideration is:
- Non-standard treasures and loot. Go beyond coins, gems, and magic items. Give out rare plants, harvested monster parts, spell components, and just generally bizarre artifacts that are maybe valuable, maybe powerful, but definitely intriguing.
- Magic is weird. Represent magic as the distortion of reality and entropic strain on time that it is. Encourage players to reskin the existing D&D spells to better suit their characters. Create new ones. Make it abberant and unsettling.
- Characters are beloved, but expendable. Make sure your players know their choices have consequences and you won't shield their characters. Also, let them retire their characters and make new ones if they want.
TL;DR O5R relies mostly on the DM's choices and flexibility to step away from vanilla 5e. They need to put in the thought and effort to apply basic old school concepts in how they run the game and make a world that's both engrossing and incredibly dangerous.