Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

Zpravodajství

Technologické zprávy

Technika

The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”

Syntax - Tasty Web Development Treats

Full Stack Developers Wes Bos and Scott Tolinski dive deep into web development topics, explaining how they work and talking about their own experiences. They cover from JavaScript frameworks like React, to the latest advancements in CSS to simplifying web tooling.

<p>What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.</p> <h2><a id="Kyle_Prinsloos_Freelancing__Beyond__Sponsor_9" name= "Kyle_Prinsloos_Freelancing__Beyond__Sponsor_9"></a>Kyle Prinsloo’s Freelancing & Beyond — Sponsor</h2> <p>Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at <a href= "https://studywebdevelopment.com/freelancing.html">https://studywebdevelopment.com/freelancing</a> and use the coupon code “syntax” at checkout to get 25% off.</p> <h2><a id="Freshbooks__Sponsor_13" name= "Freshbooks__Sponsor_13"></a>Freshbooks - Sponsor</h2> <p>Get a 30 day free trial of Freshbooks at <a href= "https://freshbooks.com/syntax">freshbooks.com/syntax</a> and put <strong>SYNTAX</strong> in the “How did you hear about us?” section.</p> <h2><a id="Show_Notes_17" name="Show_Notes_17"></a>Show Notes</h2> <p>3:05 - What Is the undocumented web?</p> <ul> <li>When traditional methods don’t give you what you want</li> <li>Wild West</li> </ul> <p>6:10 - Undocumented APIS</p> <ul> <li>Instagram’s private API</li> <li>Use dev tools’ XHR Request</li> <li>React Dev Tools will also have the data</li> <li><a href="https://syntax.fm/show/008/wes-bos-origin-story">Ep 008 - Wes Bos Origin Story</a></li> <li><a href="https://www.realtor.ca/">Realtor.ca - MLS Mapping</a></li> <li><a href="https://github.com/simonw/datasette">Simon Willison’s Zeit Day Talk about XML files</a></li> <li><a href="https://www.charlesproxy.com/">Charles Proxy App</a> - also good for debugging requests</li> </ul> <p>17:34 - Faking</p> <ul> <li>Faking User Agents <ul> <li>Sending headers from requests</li> <li>Free Blackberry wifi on plane</li> </ul> </li> </ul> <p>23:52 - Scraping</p> <ul> <li>What is craping?</li> <li><a href= "https://github.com/cheeriojs/cheerio">Cheerio</a> - jQuery like node package</li> </ul> <p>27:30 - Automation</p> <ul> <li>Cron Jobs</li> <li>Filling out forms instantly</li> <li>Headless browsers <ul> <li><a href= "https://developers.google.com/web/tools/puppeteer/">Puppeteer</a></li> <li><a href="http://phantomjs.org/">PhantomJS</a></li> </ul> </li> </ul> <p>33:30 - Proxies</p> <ul> <li>Need a fresh IP address</li> <li>Need a different country</li> <li>Side note: Canadians buying stuff in USA M4S1P1 === 41100</li> <li>API throttling</li> </ul> <p>39:00 - Tips and Tricks</p> <ul> <li>Multiple API keys</li> <li><a href= "https://www.npmjs.com/package/diskdb">diskDB</a> - Text files as databases</li> <li>Search Github for API keys</li> <li>Inspect Element and Delete Overlays</li> <li>Delete Cookiesokay</li> </ul> <p>44:45 - Stories</p> <ul> <li>Photo Voting</li> <li>MLS Listing</li> <li>Top Entrepreneur voting cheating</li> <li>Vanity Plate Checker</li> <li><a href="https://forwardcourses.com/lectures/346">My Hockey Team Sucks</a></li> <li><a href= "https://twitter.com/lawjolla/status/1008739031448875008">Vehicle auction bidding app</a></li> <li><a href="https://rg3.github.io/youtube-dl/">youtube-dl</a></li> </ul> <h2><a id="_SIIIIICK__PIIIICKS__78" name= "_SIIIIICK__PIIIICKS__78"></a>××× SIIIIICK ××× PIIIICKS ×××</h2> <ul> <li>Scott: <a href="https://amzn.to/2mvECnS">Octopath Traveler & OST</a></li> <li>Wes 1: <a href= "https://www.youtube.com/channel/UChWv6Pn_zP0rI6lgGt3MyfA">AvE Youtube Channel</a></li> <li>Wes 2: <a href= "https://www.youtube.com/channel/UCcSeeATlWJJbXpOZRYOfaDg">Mustie1 Youtube Channel</a></li> </ul> <h2><a id="Shameless_Plugs_84" name= "Shameless_Plugs_84"></a>Shameless Plugs</h2> <ul> <li><a href="https://leveluptutorials.com/pro">Scott’s Level Up Tuts Pro</a></li> <li><a href="https://wesbos.com/courses">Wes’ Courses</a></li> </ul> <h2><a id="Tweet_us_your_tasty_treats_89" name= "Tweet_us_your_tasty_treats_89"></a>Tweet us your tasty treats!</h2> <ul> <li><a href="https://www.instagram.com/stolinski/">Scott’s Instagram</a></li> <li><a href= "https://www.instagram.com/LevelUpTutorials/">LevelUpTutorials Instagram</a></li> <li><a href="https://www.instagram.com/wesbos/">Wes’ Instagram</a></li> <li><a href="https://twitter.com/wesbos">Wes’ Twitter</a></li> <li><a href="https://www.facebook.com/wesbos.developer">Wes’ Facebook</a></li> <li><a href="https://twitter.com/stolinski">Scott’s Twitter</a></li> <li>Make sure to include <a href= "https://twitter.com/SyntaxFM">@SyntaxFM</a> in your tweets</li> </ul>

What is the undocumented web? Scott and Wes dive into it, discussing APIs, faking, scraping, automation, proxies as well as tips and tricks for best practices.
 Kyle Prinsloo’s Freelancing & Beyond — Sponsor Kyle Prinsloo teaches you everything you need to know about freelancing, including how to quit your job, earn a side-income and start taking control of your life. Check it out at https://studywebdevelopment.com/freelancing and use the coupon code “syntax” at checkout to get 25% off.
 Freshbooks - Sponsor Get a 30 day free trial of Freshbooks at freshbooks.com/syntax and put SYNTAX in the “How did you hear about us?” section.
 Show Notes 3:05 - What Is the undocumented web?
 When traditional methods don’t give you what you want
 Wild West
 6:10 - Undocumented APIS
 Instagram’s private API
 Use dev tools’ XHR Request
 React Dev Tools will also have the data
 Ep 008 - Wes Bos Origin Story
 Realtor.ca - MLS Mapping
 Simon Willison’s Zeit Day Talk about XML files

Charles Proxy App - also good for debugging requests
 17:34 - Faking
 Faking User Agents Sending headers from requests
 Free Blackberry wifi on plane

 23:52 - Scraping
 What is craping?

Cheerio - jQuery like node package
 27:30 - Automation
 Cron Jobs
 Filling out forms instantly
 Headless browsers Puppeteer
 PhantomJS

 33:30 - Proxies
 Need a fresh IP address
 Need a different country
 Side note: Canadians buying stuff in USA M4S1P1 === 41100
 API throttling
 39:00 - Tips and Tricks
 Multiple API keys

diskDB - Text files as databases
 Search Github for API keys
 Inspect Element and Delete Overlays
 Delete Cookiesokay
 44:45 - Stories
 Photo Voting
 MLS Listing
 Top Entrepreneur voting cheating
 Vanity Plate Checker
 My Hockey Team Sucks
 Vehicle auction bidding app
 youtube-dl
 ××× SIIIIICK ××× PIIIICKS ××× Scott: Octopath Traveler & OST

 Wes 1: AvE Youtube Channel

 Wes 2: Mustie1 Youtube Channel

 Shameless Plugs Scott’s Level Up Tuts Pro
 Wes’ Courses
 Tweet us your tasty treats! Scott’s Instagram
 LevelUpTutorials Instagram
 Wes’ Instagram
 Wes’ Twitter
 Wes’ Facebook
 Scott’s Twitter
 Make sure to include @SyntaxFM in your tweets

Aplikace Youradio Talk

The Undocumented Web: scraping, private APIs, proxies and “alternative solutions”

Poslechněte si podcast