Creating a Notion Database for Magic: The Gathering with Python

•

So, @TomFrankly posted this tweet yesterday, & I started a quest to make a @NotionHQ database for #MagicTheGathering (@wizards_magic). Code w/ @Replit env & Template w/ empty AND filled in DBs: tinyurl.com/nmc-mtg-lmz-full Use code MAGICWAND to get it for FREE! twitter.com/TomFrankly/status/1635326820081074177

Alright, let's start. Why am I doing this in the first place? Primary reason: Introducing the ease of using python modules over javascript. Biggest problem: I do not know or care about the game at all. Spoiler: I will say, inserting 28k entries, polishing up took >24 hours.

But I do know how @NotionHQ works and how APIs work. That has to count for something, right? Right? Yeah, unfortunately I am not the gaming nerd (THE only game I have played is Among Us). And I do not plan to be one. This is a real-time account of the process. Incoming: 🧵

Step 1: Creating a @Replit That is the simplest thing you would do here. i. Sign up for a repl account. ii. Create a python repl. iii. Create a notion integration and get the API key iv. Create a secret variable in repl named `notion_api_key`

Step 2: Finding python libraries I know I am going to use notional (github.com/jheddings/notional) by @jheddings. I have used it in over 50 personal projects now (and this is what NOMs are made on | shameless plug, follow to be notified when released: producthunt.com/products/noms-2)

Finding Magic The Gathering library. Let's go to point 1. I have no idea what this thing is, but I like playing around with APIs and Notion. If I were an MTG enthusiast, I'd first create a @NotionHQ DB. But I have no idea what it would entail. So, we look at libraries first.

I found two of them: 1. MTG SDK (github.com/MagicTheGathering/mtg-sdk-python) 2. Scryfall (scryfall.com/docs/api/images) Now, one thing you always need to remember is, if something is nerdy enough, someone will create a python wrapper for it. And voila, Scrython: github.com/NandaScott/Scrython

Now you might ask, what prompted me to look for a non-official library in the first place? That would be the useless `image_url` property sent through the official one. Why is it useless? Because Notion API does not allow you to upload files, and, MTG provides .ashx urls.

Now Cat, why don't you just use the `multiverse_id` provided by the official API and use it to retrieve image from `scrython`. You wish. But scrython has its own card imagery URIs that are NOT DIRECTLY RELATED to the `multiverse_id`.

Whatever. We are going to add all of them and figure it out as we go. So, go to the repl we created and add 3 packages in the Shell tab: `poetry add notional poetry add scrython poetry add mtgsdk` This would end up with your `pyproject.toml` looking like this.

Step 2: One thing that MTG API Docs (docs.magicthegathering.io/) does do, is provide the properties I need to add to the Notion Database. I have two Notion databases based on the return types in the API docs. We also create 2 new repl secrets: `cards_db_id` and `sets_db_id`.

Why 2 databases? Well, i. What you are building here is a lot more complex than Pokedex, not because of size but because I decided to have a relation to sets database. ii. You can make this all one database if you want to, but then, that ain't the point of @NotionHQ, is it?

Step 3: Getting the damn image URL because it seems like the card image is the most important part 15 minutes later, I have figured out the image process that I would likely use. Combine card set code and number from the official API to get the image scryfall.com/docs/api/cards/collector

You didn't think this mess was solved, did ya? Nothing is solved this easily, if it were, we wouldn't be needing rubber ducks to solve out problems (en.wikipedia.org/wiki/Rubber_duck_debugging) So, apparently MTG has multi-faced cards, and we need to handle them. scryfall.com/docs/api/cards#card-face-objects

Step 4: Putting the basic python code together i. In this step, we will try to insert every card information, AS RECEIVED, into the notion database. We do not care about images or sets at the moment. ii. Yes, we would debug it later, but this is how you should go about it too.

💡 Tips: Remember to share your database with the integration you created. If you get error 503 with official MTG API, just try again. If you want to know the properties of an object in python (because MTG official API returns a card obejct), use `dir(obj)`.

Suddenly, I am no longer liking the idea of using MTG API. Why? Because it keeps throwing service unavailable errors. Now, I am going to give it 25 more minutes, while I figure out image URLs to see if it is up or not. If it isn't, I am probably just going to shift to scrython.

Oh hey, we have hit a block again. This time, we find out that scrython does not support /cards/:code/:number(/:lang) and we need to find another way to get the image URLs. Pros: More time for the MTG API to NOT BE unavailable Cons: Everything else

Alright, here is to hoping that MTG and Scryfall have the same naming conventions for their cards. Because, now we are retrieving cards using `scrython.cards.Named(fuzzy=card.name)` Why not use scrython in the first place? This is called code debt.

40 minutes later and the MTG SDK is still not up. I guess we are now tackling the code debt & using scrython. Goddamnit. Hey at least now we do not need to figure out image URIs separately? Now, here is to hoping we do not need to change the properties in @NotionHQ DB.

For rate limiting, we will be using `time.sleep(0.1)`. Everyone, remember, no complicated systems for 3 hour builds. Now onto to the basic question for which I was initially using MTG SDK, how do I get all the cards without using the bulk data endpoint in scryfall?

I give up on trying to retrieve fresh data. Why was I doing so in the first place? I don't know. I will be using the bulk data here: scryfall.com/docs/api/bulk-data Now, I know @TomFrankly says 22630 cards but uh, the closest match was "Oracle Cards" of 28173. We will roll with that!

See, now that we have this, the easiest thing to do would obviously be to: (i) convert this json to csv (ii) upload csv to notion (iii) fix image file imports somehow But that doesn't teach you about API, does it? And, I do not want to be fixing csv imports in @NotionHQ.

So, we go back to the Step 2 and figure out if there are different properties in scryfall API as compared to MTG API. And I bet you 50$ there are. We specifically add keywords, oracle text and oracle id. I am making sure all the existing properties I mapped in the code exist.

Well, multiple deleted properties (15 for the cards database, 9 for the sets database) and added properties (5 for the cards database, 4 for the sets database) later, we are getting something done! YAY! I would have used set's `icon_svg_uri` but it says the URI might change.

Given this is a "fun" project, I am not looking at edge cases. What if the bulk data does not return oracle cards at index 0? Yeah, we cross our fingers and hope it does 🤣 I think the sets insertion is now working (sped up 5x in the gif) Now to hoping that the cards work!

Oh well 30 minutes later: `repl process died unexpectedly: signal: killed` This time we are pickling the damn set dictionary so that we do not delete & add the same set multiple times when debugging the card addition code. Some array based error bashing later, we see the END 🤯

NOPE. Not yet. The bulk API json does not come with power, toughness, life modifiers etc for all cards. It is not just the data that might be missing. IT IS THE WHOLE DAMN KEY! AWESOME! I tried setting it to `types.TYPE[None]` if the info is missing, but that doesn't work! 😠

Hello everyone. It works! Some finicking with the file object, set relations and it seems like everything is on track. We are almost 6 hours into this project at this time. I think it is final run time while I go and sleep (it is almost 5 am)

Oh well, we keep running into this pesky Notion 502 error issue. The best part? This error is completely absent from their developer page: developers.notion.com/reference/status-codes I need to add some form of already added persistence for sets and cards. SIGH. 😅 Doing that right now.

⚠️ I also changed 'Released At' type to Date. If you do not ADHD, you do not get this. I cannot do anything else because this small thing keeps nagging me 7 hours later. Of course you can fill in the db & change it later to date and it will work. But I cannot let it be 😔

I have downloaded this onto my computer to run it through night. A very naive parallelized version (break the list into 2500 cards and run it 10 times) along with `set_dict` pickle dumped from replit. This will be the end of the process if it ends up running without errors.

It is 6:30am and I should sleep. No, you will not learn about exponential backoff because this is not a tutorial. Especially because I have 2 paper submissions, 4 meetings and 3 reading groups today! YAY! Does it work though? 😉 (Spoiler: YES, it works, MANY Error 502 later).

8 HOURS LATER! Code w/ @Replit env & Template w/ empty AND filled in DBs: tinyurl.com/nmc-mtg-lmz-full Use code MAGICWAND to get it for FREE! If you find it useful for learning Notion API in Python or building your MTG collection, PLEASE buy me a coffee: buymeacoffee.com/nerdymomocat

I share weird notion tips that work every weekend. Megathread here: twitter.com/nerdymomocat/status/1634569897446608896 Megathread of academic workflows shared on weekdays here: twitter.com/nerdymomocat/status/1632321456968437760 Get NOMs & MEOWs: automations for Notion & other productivity software, here: nerdymomocat.lemonsqueezy.com/