Create instagram_crawler.py #2508

yogeshwaran01 · 2020-09-29T09:46:43Z

An algorithm crawls the Instagram page of the user and Scarpe the data.

Describe your change:

Add an algorithm?
Fix a bug or typo in an existing algorithm?
Documentation change?

Checklist:

It crawls the Instagram page of the user and Scarpe the data.

cclauss

Cool submission!!

cclauss · 2020-09-29T10:02:13Z

web_programming/instagram_crawler.py

+from bs4 import BeautifulSoup
+import json
+
+headers = \


No backslashes please in Python code. See PEP8. The problem with backslashes is that whitespace to the right of the backslash breaks the script on a change that is invisible to the reader.

You can run your code through psf/black to autofix that issue as discussed in CONTRIBUTING.md

web_programming/instagram_crawler.py

cclauss · 2020-09-29T10:04:34Z

web_programming/instagram_crawler.py

+
+    def __init__(self, username):
+        self.username = username
+        self.url = 'https://www.instagram.com/{}/'.format(username)


Suggested change

self.url = 'https://www.instagram.com/{}/'.format(username)

self.url = f'https://www.instagram.com/{username}/'

As discussed in CONTRIBUTING.md, please use f-strings where they make sense.

Do we really want to query the backend (Instagram) for every one of the fields below? Most of these values are very static and are unlikely to change minute-to-minute. Perhaps it would be better to have a self.user_data dict that contained the results of self.get_json() and the other methods could use that data.

cclauss · 2020-09-29T10:06:39Z

web_programming/instagram_crawler.py

+            info = html_1(soup)
+            return info
+        except:
+            info = html_2(soup)
+            return info


Suggested change

info = html_1(soup)

return info

except:

info = html_2(soup)

return info

return html_1(soup)

except: # <-- This repo does not accept bare excepts

return html_2(soup)

Bare excepts are discussed in PEP8 and https://realpython.com/the-most-diabolical-python-antipattern/

cclauss · 2020-09-29T10:22:06Z

web_programming/instagram_crawler.py

+            info = html_2(soup)
+            return info
+
+    def get_followers(self):


Let's make this (and similar methods below) into a @property so that we can use the syntax instagram_user.number_of_followers (without the ()).

Suggested change

def get_followers(self):

@property

def number_of_followers(self) -> int:

Also, let's streamline to a one-line implementation like:

return self.get_json()['edge_followed_by']['count'] # or... return self.data['edge_followed_by']['count']

In general, do not create a variable that you get rid of on the very next line unless the variable name really helps the reader understand something nonobvious.

cclauss · 2020-09-29T10:22:49Z

web_programming/instagram_crawler.py

+        followers = info['edge_followed_by']['count']
+        return followers
+
+    def get_followings(self):


Suggested change

def get_followings(self):

def get_number_of_followings(self) -> int:

cclauss · 2020-09-29T10:23:27Z

web_programming/instagram_crawler.py

+        following = info['edge_follow']['count']
+        return following
+
+    def get_posts(self):


Suggested change

def get_posts(self):

def get_number_of_posts(self) -> int:

cclauss · 2020-09-29T10:23:50Z

web_programming/instagram_crawler.py

+        posts = info['edge_owner_to_timeline_media']['count']
+        return posts
+
+    def get_biography(self):


Suggested change

def get_biography(self):

def get_biography(self) -> str:

cclauss · 2020-09-29T10:28:08Z

web_programming/instagram_crawler.py

+user = Instagram('github')
+print(user.is_verified())
+print(user.get_biography())


Code that is at global scope will be run by our Travis CI / pytest process as discussed in CONTRIBUTING.md.

Suggested change

user = Instagram('github')

print(user.is_verified())

print(user.get_biography())

if __name__ == '__main__':

user = Instagram('github')

print(f"{user.is_verified() = })

print(f"{user.get_biography() = })

Co-authored-by: Christian Clauss <[email protected]>

yogeshwaran01

According to @cclauss some changes are done

cclauss · 2020-09-29T11:54:11Z

web_programming/instagram_crawler.py

-    {
-        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
+headers = {
+    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"


Suggested change

"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) "

"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

TravisBuddy · 2020-09-29T12:13:32Z

Hey @yogeshwaran01,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 303e78f0-024d-11eb-aba2-872ffb2742c8

TravisBuddy · 2020-09-29T12:13:32Z

Hey @yogeshwaran01,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 303bb9d0-024d-11eb-aba2-872ffb2742c8

TravisBuddy · 2020-09-29T12:13:33Z

Hey @yogeshwaran01,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 303f1530-024d-11eb-aba2-872ffb2742c8

TravisBuddy · 2020-09-29T12:13:33Z

Hey @yogeshwaran01,
Something went wrong with the build.

TravisCI finished with status errored, which means the build failed because of something unrelated to the tests, such as a problem with a dependency or the build process itself.

View build log

TravisBuddy Request Identifier: 30424980-024d-11eb-aba2-872ffb2742c8

cclauss · 2020-09-29T12:22:20Z

Closing in favor of #2509

Create instagram_crawler.py

6428bf5

It crawls the Instagram page of the user and Scarpe the data.

cclauss requested changes Sep 29, 2020

View reviewed changes

yogeshwaran01 and others added 3 commits September 29, 2020 16:38

Update instagram_crawler.py

7e359cf

changed class name to InstagramUser

8bc9104

Co-authored-by: Christian Clauss <[email protected]>

some chamges had done

bafda76

yogeshwaran01 commented Sep 29, 2020

View reviewed changes

cclauss reviewed Sep 29, 2020

View reviewed changes

cclauss closed this Sep 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create instagram_crawler.py #2508

Create instagram_crawler.py #2508

yogeshwaran01 commented Sep 29, 2020

cclauss left a comment

cclauss Sep 29, 2020

cclauss Sep 29, 2020

cclauss Sep 29, 2020 •

edited

Loading

cclauss Sep 29, 2020

cclauss Sep 29, 2020

cclauss Sep 29, 2020

cclauss Sep 29, 2020

cclauss Sep 29, 2020

cclauss Sep 29, 2020

cclauss Sep 29, 2020

yogeshwaran01 left a comment

cclauss Sep 29, 2020

TravisBuddy commented Sep 29, 2020

TravisBuddy commented Sep 29, 2020

TravisBuddy commented Sep 29, 2020

TravisBuddy commented Sep 29, 2020

cclauss commented Sep 29, 2020

	self.url = 'https://www.instagram.com/{}/'.format(username)
	self.url = f'https://www.instagram.com/{username}/'

	def get_followers(self):
	@property
	def number_of_followers(self) -> int:

	def get_followings(self):
	def get_number_of_followings(self) -> int:

	"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
	"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) "
	"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

Create instagram_crawler.py #2508

Create instagram_crawler.py #2508

Conversation

yogeshwaran01 commented Sep 29, 2020

Describe your change:

Checklist:

cclauss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cclauss Sep 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yogeshwaran01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TravisBuddy commented Sep 29, 2020

TravisBuddy Request Identifier: 303e78f0-024d-11eb-aba2-872ffb2742c8

TravisBuddy commented Sep 29, 2020

TravisBuddy Request Identifier: 303bb9d0-024d-11eb-aba2-872ffb2742c8

TravisBuddy commented Sep 29, 2020

TravisBuddy Request Identifier: 303f1530-024d-11eb-aba2-872ffb2742c8

TravisBuddy commented Sep 29, 2020

TravisBuddy Request Identifier: 30424980-024d-11eb-aba2-872ffb2742c8

cclauss commented Sep 29, 2020

cclauss Sep 29, 2020 •

edited

Loading