Wes Bos & Scott Tolinski - Full Stack JavaScrip...

Zpravodajství

Technologické zprávy

Technika

763: Web Scraping + Reverse Engineering APIs

Syntax - Tasty Web Development Treats

Full Stack Developers Wes Bos and Scott Tolinski dive deep into web development topics, explaining how they work and talking about their own experiences. They cover from JavaScript frameworks like React, to the latest advancements in CSS to simplifying web tooling.

Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and techniques to scrape (almost) any website with ease. <h3 class="code-line" data-line-start="2" data-line-end="3"><a id= "Show_Notes_2"></a>Show Notes</h3> <ul> <li class="has-line-data" data-line-start="4" data-line-end="5"> <a href="#t=00:00">00:00</a> Welcome to Syntax!</li> <li class="has-line-data" data-line-start="5" data-line-end="6"> <a href="#t=03:13">03:13</a> Brought to you by <a href="https://sentry.io/syntax">Sentry.io</a>.</li> <li class="has-line-data" data-line-start="6" data-line-end="7"> <a href="#t=05:00">05:00</a> What is scraping?</li> <li class="has-line-data" data-line-start="7" data-line-end="8"> <a href="https://socialblade.com/">Socialblade</a>.</li> <li class="has-line-data" data-line-start="8" data-line-end="9"> <a href="#t=08:01">08:01</a> Examples of past scrapers.</li> <li class="has-line-data" data-line-start="9" data-line-end="10"> <a href="https://www.canadiantire.ca/en.html">Canadian Tire</a>.</li> <li class="has-line-data" data-line-start="10" data-line-end="11"> <a href="#t=10:06">10:06</a> Cloud app downloader.</li> <li class="has-line-data" data-line-start="11" data-line-end="12"> <a href="#t=16:13">16:13</a> Other use cases.</li> <li class="has-line-data" data-line-start="12" data-line-end="13"> <a href="#t=16:58">16:58</a> Scraping 101.</li> <li class="has-line-data" data-line-start="13" data-line-end="14"> <a href="#t=17:28">17:28</a> Client Side.</li> <li class="has-line-data" data-line-start="14" data-line-end="15"> <a href="#t=19:08">19:08</a> Private API.</li> <li class="has-line-data" data-line-start="15" data-line-end="16"> <a href="https://proxyman.io/iphone-device">Proxyman</a>.</li> <li class="has-line-data" data-line-start="16" data-line-end="17"> <a href="#t=22:40">22:40</a> Server rendered.</li> <li class="has-line-data" data-line-start="17" data-line-end="18"> <a href="#t=23:27">23:27</a> Initial state.</li> <li class="has-line-data" data-line-start="18" data-line-end="19"> <a href="#t=24:57">24:57</a> What format is the data in?</li> <li class="has-line-data" data-line-start="19" data-line-end="20"> <a href="https://developer.chrome.com/docs/puppeteer">Google Puppeteer Extension</a>.</li> <li class="has-line-data" data-line-start="20" data-line-end="21"> <a href="#t=27:08">27:08</a> Working with the DOM.</li> <li class="has-line-data" data-line-start="21" data-line-end="22"> <a href="#t=27:12">27:12</a> <a href= "https://www.npmjs.com/package/linkedom">Linkedom npm package</a>.</li> <li class="has-line-data" data-line-start="22" data-line-end="23"> <a href="#t=29:02">29:02</a> <a href= "https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll"> querySelector</a> everything.</li> <li class="has-line-data" data-line-start="23" data-line-end="24"> <a href="#t=31:28">31:28</a> How to find the elements without classes.</li> <li class="has-line-data" data-line-start="24" data-line-end="25"> <a href="#t=34:08">34:08</a> Use <a href= "https://www.w3schools.com/xml/xpath_syntax.asp">XPath</a> selectors for select by word.</li> <li class="has-line-data" data-line-start="25" data-line-end="26"> <a href="#t=34:53">34:53</a> Make them as flexible as you can. Classes change!</li> <li class="has-line-data" data-line-start="26" data-line-end="27"> <a href="#t=35:10">35:10</a> AI is good at this!</li> <li class="has-line-data" data-line-start="27" data-line-end="28"> <a href="#t=36:26">36:26</a> File downloading.</li> <li class="has-line-data" data-line-start="28" data-line-end="29"> <a href="#t=38:20">38:20</a> Working with protected routes.</li> <li class="has-line-data" data-line-start="29" data-line-end="30"> <a href="https://proxyman.io/">Proxyman</a>.</li> <li class="has-line-data" data-line-start="30" data-line-end="31"> <a href="#t=40:41">40:41</a> Programatically retrieve authentication keys because they are short-lived.</li> <li class="has-line-data" data-line-start="31" data-line-end="32"> <a href="https://www.npmjs.com/package/fetch-cookie">Fetch Cookie</a>.</li> <li class="has-line-data" data-line-start="32" data-line-end="33"> <a href="#t=43:20">43:20</a> Deal-breakers.</li> <li class="has-line-data" data-line-start="33" data-line-end="34"> <a href="https://en.wikipedia.org/wiki/Mechanical_Turk">Mechanical Turk</a>.</li> <li class="has-line-data" data-line-start="34" data-line-end="35"> <a href="#t=44:58">44:58</a> What happened with Amazon?</li> <li class="has-line-data" data-line-start="35" data-line-end="36"> <a href= "https://www.wsj.com/articles/uniqlos-parent-company-bets-big-on-tiny-rfid-chips-600b124f"> Uniqlo Self-Checkout</a></li> <li class="has-line-data" data-line-start="36" data-line-end="37"> <a href="#t=46:42">46:42</a> Wes’ portable refrigerator utopia.</li> <li class="has-line-data" data-line-start="37" data-line-end="39"> <a href="#t=47:25">47:25</a> Sick Picks & Shameless Plugs.</li> </ul> <h3 class="code-line" data-line-start="39" data-line-end="40"> <a id="Sick_Picks_39"></a>Sick Picks</h3> <ul> <li class="has-line-data" data-line-start="41" data-line-end="42"> Scott: <a href= "https://folivora.ai/keyboardcleantool">KeyboardCleanTool</a>.</li> <li class="has-line-data" data-line-start="42" data-line-end="44"> Wes: <a href= "https://github.com/koekeishiya/yabai/wiki">Yabai</a>.</li> </ul> <h3 class="code-line" data-line-start="44" data-line-end="45"> <a id="Shameless_Plugs_44"></a>Shameless Plugs</h3> <ul> <li class="has-line-data" data-line-start="46" data-line-end="48"> Scott: <a href="https://www.youtube.com/@syntaxfm">Syntax on YouTube</a></li> </ul> <h3 class="code-line" data-line-start="48" data-line-end="49"> <a id="Hit_us_up_on_Socials_48"></a>Hit us up on Socials!</h3> Syntax: <a href="https://twitter.com/syntaxfm">X</a> <a href= "https://www.instagram.com/syntax_fm/">Instagram</a> <a href= "https://www.tiktok.com/@syntaxfm">Tiktok</a> <a href= "https://www.linkedin.com/company/96077407/admin/feed/posts/">LinkedIn</a> <a href="https://www.threads.net/@syntax_fm">Threads</a> Wes: <a href="https://twitter.com/wesbos">X</a> <a href= "https://www.instagram.com/wesbos/">Instagram</a> <a href= "https://www.tiktok.com/@wesbos">Tiktok</a> <a href= "https://www.linkedin.com/in/wesbos/">LinkedIn</a> <a href= "https://www.threads.net/@wesbos">Threads</a> Scott:<a href="https://twitter.com/stolinski">X</a> <a href= "https://www.instagram.com/stolinski/">Instagram</a> <a href= "https://www.tiktok.com/@stolinski">Tiktok</a> <a href= "https://www.linkedin.com/in/stolinski/">LinkedIn</a> <a href= "https://www.threads.net/@stolinski">Threads</a> Randy: <a href="https://twitter.com/randyrektor">X</a> <a href= "https://www.instagram.com/randyrektor/">Instagram</a> <a href= "https://www.youtube.com/@randyrektor">YouTube</a> <a href= "https://www.threads.net/@randyrektor">Threads</a>

Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and techniques to scrape (almost) any website with ease.
 Show Notes  00:00 Welcome to Syntax!
 03:13 Brought to you by Sentry.io.
 05:00 What is scraping?
 Socialblade.
 08:01 Examples of past scrapers.
 Canadian Tire.
 10:06 Cloud app downloader.
 16:13 Other use cases.
 16:58 Scraping 101.
 17:28 Client Side.
 19:08 Private API.
 Proxyman.
 22:40 Server rendered.
 23:27 Initial state.
 24:57 What format is the data in?
 Google Puppeteer Extension.
 27:08 Working with the DOM.
 27:12 Linkedom npm package.
 29:02 querySelector everything.
 31:28 How to find the elements without classes.
 34:08 Use XPath selectors for select by word.
 34:53 Make them as flexible as you can. Classes change!
 35:10 AI is good at this!
 36:26 File downloading.
 38:20 Working with protected routes.
 Proxyman.
 40:41 Programatically retrieve authentication keys because they are short-lived.
 Fetch Cookie.
 43:20 Deal-breakers.
 Mechanical Turk.
 44:58 What happened with Amazon?
  Uniqlo Self-Checkout

 46:42 Wes’ portable refrigerator utopia.
 47:25 Sick Picks & Shameless Plugs.
  Sick Picks  Scott: KeyboardCleanTool.
 Wes: Yabai.
  Shameless Plugs  Scott: Syntax on YouTube

  Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads
 Wes: X Instagram Tiktok LinkedIn Threads
 Scott:X Instagram Tiktok LinkedIn Threads
 Randy: X Instagram YouTube Threads

Aplikace Youradio Talk

763: Web Scraping + Reverse Engineering APIs

Poslechněte si podcast