Web Scraping With JavaScript Using Axios and JSDom
It's been almost a year since I created a website whose primary purpose was to consume a Japanese idiom API and give users a new idiom (and its pronunciation) each time they visited in a new browser window. When I originally imagined a concept for this website, I wanted to offer users an idiom, its pronunciation, and its definition.
ReadMigrating from MongoDb to PostgreSQL (Case Insensitive Sorting)
After creating a new database in PostgreSQL, I needed to connect my Node/Express application to my database. I decided to use node-postgres to do this. I followed the recommended setup for node-postgres and created a new db folder with an index.js file setting up my PostgreSQL database. After quickly testing to ensure that I was connected to the database, I began the process of updating my routes to reflect the new database.
ReadMigrating from MongoDb to PostgreSQL (Database Structure)
As I continued to restructure my database to accommodate user authentication, I ran into several issues. I solved some of these problems using embedded documents and aggregation. However, as I continued to learn more about databases, I came to an important realization: I misunderstood the relationships within the data I created; I incorrectly identified various relationships within my database as one-to-many.
ReadUsing Aggregation to Work With Nested Arrays in MongoDB
After restructuring my database to accommodate user authentication, I was faced with a new challenge: displaying and manipulating information from arrays.
ReadCreating One-to-Many Relationships with Embedded Documents in MongoDB
After adding user authentication, I had three separate schemas for my application: a user schema, a video schema, and a list schema (that referenced videos created using the video schema). After creating an account and logging in, users could see any video or list that any user had created. However, users should have only been able to see the videos and lists that they themselves added. I had to create some sort of relationship between the user and videos and lists of videos.
ReadGetting TikTok Date Information in Node.js
One of the features many TikTok users, including myself, would like to have is the ability to search for or sort videos according to when they were uploaded. This information is not included in the official TikTok API, so I assumed that it was not available. However, I recently read a very helpful post from Digital Forensics that detailed how to get timestamps from TikTok videos. I decided to implement the steps in Node.js and append this information to videos saved in the database.
ReadMaking Many HTTP Requests Using Recursion
One of the most important functions for my TikTok bookmark app is the ability to import an entire list of liked videos. During the process of testing files for liked videos, I realized that longer lists (with thousands of videos) struggled to import all files in one attempt. After examining 150 videos or so, I started receiving error messages about the connect timing out. I fiddled with the asynchronous code I created (and upgraded the for loop I was using to an ES6 for...of loop in the process), but I continued to see the same error messages after a minute or so of processing HTTP get requests.
Read