Category: Programming

Summary: You can caculate the logistic sigmoid function in Python using:

• The Math Module: 1 / (1 + math.exp(-x))
• The Numpy Library: 1 / (1 + np.exp(-x))
• The Scipy Library: scipy.special.expit(x)

---

Problem: Given a logistic sigmoid function:

If the value of x is given, how will you calculate F(x) in Python? Let’s say x=0.458.

Note: Logistic sigmoid function is defined as (1/(1 + e^-x)) where x is the input variable and represents any real number. The function returns a value that lies within the range -1 and 1. It forms an S-shaped curve when plotted on a graph.

❒Method 1: Sigmoid Function in Python Using Math Module

Approach: Define a function that accepts x as an input and returns F(x) as 1/(1 + math.exp(-x)).

Code:

import math def sigmoid(x): return 1 / (1 + math.exp(-x)) print(sigmoid(0.458)) # OUTPUT: 0.6125396134409151

Caution: The above solution is mainly intended as a simple one-to-one translation of the given sigmoid expression into Python code. It is not strictly tested or considered to be a perfect and numerically sound implementation. In case you need a more robust implementation, some of the solutions to follow might prove to be more instrumental in solving your case.

Here’s a more stable implementation of the above solution:

import math def sigmoid(x): if x >= 0: k = math.exp(-x) res = 1 / (1 + k) return res else: k = math.exp(x) res = k / (1 + k) return res print(sigmoid(0.458))

Note: exp() is a method of the math module in Python that returns the value of E raised to the power of x. Here, x is the input value passed to the exp() function, while E represents the base of the natural system of the logarithm (approximately 2.718282).

❒Method 2: Sigmoid Function in Python Using Numpy

The sigmoid function can also be implemented using the exp() method of the Numpy module. numpy.exp() works just like the math.exp() method, with the additional advantage of being able to handle arrays along with integers and float values.

Let’s have a look at an example to visualize how to implement the sigmoid function using numpy.exp()

import numpy as np def sigmoid(x): return 1 / (1 + np.exp(-x)) print(sigmoid(0.458)) # OUTPUT: 0.6125396134409151

Probably a more numerically stable version of the above implementation is as follows:

import numpy as np def sigmoid(x): return np.where(x < 0, np.exp(x) / (1 + np.exp(x)), 1 / (1 + np.exp(-x))) print(sigmoid(0.458)) # OUTPUT: 0.6125396134409151

#Example 2: Let’s have a look at an implementation of the sigmoid function upon an array of evenly spaced values with the help of a graph in the following example.

import numpy as np
import matplotlib.pyplot as plt def sigmoid(x): return np.where(x < 0, np.exp(x) / (1 + np.exp(x)), 1 / (1 + np.exp(-x))) val = np.linspace(start=-10, stop=10, num=200)
sigmoid_values = sigmoid(val)
plt.plot(val, sigmoid_values)
plt.xlabel("x")
plt.ylabel("sigmoid(X)")
plt.show()

Output:

Explanation:

• Initially, we created an array of evenly spaced values within the range of -10 and 10 with the help of the linspace method of the Numpy module, i.e., val.
• We then used the sigmoid function on these values. If you print them out, you will find that they are either extremely close to 0 or very close to 1. This can also be visualized once the graph is plotted.
• Finally, we plotted the sigmoid function graph that we previously computed with the help of the function. The x-axis maps the values contained in val, while the y-axis maps the values returned by the sigmoid function.

---

Do you want to become a NumPy master? Check out our interactive puzzle book Coffee Break NumPy and boost your data science skills! (Amazon link opens in new tab.)

❒Method 3: Sigmoid Function in Python Using the Scipy Library

Another efficient way to calculate the sigmoid function in Python is to use the Scipy libraries expit function.

Example 1: Calculating logistic sigmoid for a given value

from scipy.special import expit
print(expit(0.458)) # OUTPUT: 0.6125396134409151

Example 2: Calculating logistic sigmoid for multiple values

from scipy.special import expit
x = [-2, -1, 0, 1, 2]
for value in expit(x): print(value)

Output:

0.11920292202211755
0.2689414213699951
0.5
0.7310585786300049
0.8807970779778823

Recommended Read: Logistic Regression in Python Scikit-Learn

❒Method 4: Transform the tanh function

Another workaround to compute the sigmoid function is to transform the tanh function of the math module as shown below:

import math sigmoid = lambda x: .5 * (math.tanh(.5 * x) + 1)
print(sigmoid(0.458)) # OUTPUT: 0.6125396134409151

Since, mathematically sigmoid(x) == (1 + tanh(x/2))/2. Hence, the above implementation should work and is a valid solution. However, the methods mentioned earlier are undoubtedly more stable numerically and superior to this solution.

Conclusion

Well, that’s it for this tutorial. We have discussed as many as four ways of calculating the logistic sigmoid function in Python. Feel free to use the one that suits your requirements.

I hope this article has helped you. Please subscribe and stay tuned for more interesting solutions and tutorials. Happy learning!

---

TensorFlow – A Hands-On Introduction to Deep Learning and Neural Networks for Beginners

This course gives you a charming introduction into deep learning and neural networks using Google’s TensorFlow library for Python beginners.

The post is part 6 and continues our Smart Contract Security Series:

• Ownership Exploit
• Private Variable Exploit
• Reentrancy Attack
• tx.origin Phishing Attack
• Denial of Service Attack
• Storage Collision Attack

The DelegateCall attack or storage collision is expounded in this post.

Before you can grasp this exploit, you must first understand how Solidity saves state variables as explained here. 

We start with the differences between call and delegatecall in Solidity, followed by exploiting the vulnerability of the delegatecall using the proxy contracts (mostly in smart contract upgrades), and then a solution for the attack.

Let’s start the journey!

Call VS DelegateCall

Solidity supports two low-level interfaces for interaction or sending messages to the contract functions.

These interfaces operate on addresses rather than contract instances (using this keyword). The key differences are highlighted with an example.

Call

It allows you to call the code of the callee contract from the caller with the storage context of the callee.

In order to understand this confusing sentence, let’s consider two contracts A, and CallA, with the naming convention as below:

• A is the callee,
• CallA is the caller

// Callee
contract A
{ uint256 public x; function foo(uint256 _x) public { x = _x; }
} // Caller
contract CallA
{ uint256 public x; function callfoo(address _a) public { (bool success,) = _a.call(abi.encodeWithSignature("foo(uint256)", 15)); require(success, "Call was not successful"); }
}

To test, deploy the contracts on Remix, and when you execute the caller (CallA -> callfoo), you can verify that foo() gets called, and the value of ‘x‘ in the callee(A ->x) is set to 15.

Note: It is also possible to send Ether and gas as part of the call using value and gas as params.

The above scenario is described in the figure as shown.

DelegateCall

It allows you to call the code of the callee contract from the caller with the storage context of the caller.

As previously mentioned, let’s consider two contracts A and DelegateCallA, with the naming convention as below:

• A is the callee,
• DelegateCallA is the caller

contract A
{ uint256 public x; function foo(uint256 _x) public { x = _x; }
} contract DelegateCallA
{ uint256 public x; function callfoo(address _a) public { (bool success,) = _a.delegatecall(abi.encodeWithSignature("foo(uint256)", 15)); require(success, "Delegate Call was not successful"); }
}

To test, deploy the contracts on Remix, and when you execute the caller (DelegateCallA -> ‘callfoo’), you can verify that foo() gets called and the value of ‘x‘ in the callee(A ->x) is still 0, while the value of x in the caller (DelegateCallA -> x) is 15.

Equipped with the above examples, it is evident that the delegatecall, executes in the caller’s context, while the call executes in the callee context.

A picture speaks a thousand words. The above scenario is in the below figure.

One use case of call is the transfer of Ether to a contract, and it passes all the gas to the receiving function, while the use cases of the delegatecall are when a contract invokes a library with public functions or uses a proxy contract to write smart upgradeable contracts.

Exploit with delegatecall

The most widely adopted technique to upgrade contracts is utilizing a proxy contract.

A proxy interposes the actual logical contract and the dapp interface. To update the logical contract with a new version (say V2), only the new deployed address of the logical contract is passed to the proxy.

This helps achieve minimal or no changes in the dapp/web3 interface, saving a lot of development time.

Let us write a quick and short proxy, and a logical contract (say V1). For the same, create a file DelegateCall.sol with Proxy and V1 contracts as below.

contract Proxy
{ uint256 public x; address public owner; address public logicalAddr; constructor(address _Addr) { logicalAddr = _Addr; owner = msg.sender; } function upgrade(address _newAddr) public { logicalAddr = _newAddr; } // To call any function of the logical contract fallback() external payable { (bool success, ) = logicalAddr.delegatecall(msg.data); require(success , " Error calling logical contract"); }
}

V1, This represents version 1 of the logical contract.

contract V1
{ uint256 public x; // abi.encodedWithSignature("increment_X()") = 0xeaf2926e function increment_X() public { x += 1; }
}

Compile, deploy and run the contracts in Remix with the constructor param in proxy as the address of the V1 contract.

You can observe that, when the abi.encodedWithSignature("increment_X()"))  is passed as calldata to Proxy (fallback() is triggered), the function increment_X() in V1 is called.

Abi encoding is calculated using the tool,

$ npm install web3-eth-abi

and then

const Web3EthAbi = require('web3-eth-abi'); > Web3EthAbi.encodeFunctionSignature("increment_X()") '0xeaf2926e'

As discussed above in delegatecall, the storage context of the caller (i.e., Proxy) is used, and the value of x in Proxy is incremented by 1.

So far, this is all good.

At some point in the future, it is decided to upgrade the V1 contract with new functionality, let’s call it V2.

Create a new contract V2

contract V2
{ uint256 public x; uint256 public y; function increment_X() public { x += 1; } // abi.encodedWithSignature("set_Y(uint256)", 10) //0x1675b4f5000000000000000000000000000000000000000000000000000000000000000a function set_Y(uint256 _y) public { y = _y; }
}

Compile and deploy V2.

Pass the address of V2, to upgrade() in Proxy as V2 is the new contract we need.

When abi.encodedWithSignature("set_Y(uint256)", 10)) is passed as calldata to proxy, the function increment_Y() in V2 is called.

The value of y is 10, but wait a minute, surprise, surprise!

As there is no y in the Proxy contract, and as the storage context of Proxy is used, it has overwritten the second param in Proxy (i.e., owner) with 10 (or 0x000000000000000000000000000000000000000A).

With the owner address changed, the attacker is now in complete control of all the contracts.

How to Prevent the Attack

The delegatecall is tricky to use, and erroneous usage might have disastrous consequences.

For example, possible solutions to the above problem can be

1. If possible, avoid using additional storage variables or go stateless in the upgraded contract – V2.
2. Mirror the storage layout in V2, in other words, the contract calling delegatecall and the contract being called must have the same storage layout.
3. By implementing unstructured storage in proxy with the help of assembly code as in OpenZeppelins proxy and not having any storage variables in proxy apart from the logical contract address.

Outro

In this tutorial, we saw how delegatecall can lead to disastrous results with an incorrect understanding or usage.

While using delegatecall, it is vital to keep it in our minds that delegatecall keeps context intact (storage, caller, etc…).

Even though there are certain problems associated with delegatecall, it is very often used in many contracts such as OpenZeppelin, Solidity libraries, EIP2535 diamonds, and many more.

To conclude, use delegatecall, but with care!

---

Summary: The most straightforward way to overwrite the previous print to stdout is to set the carriage return ('\r') character within the print statement as print(string, end = "\r"). This returns the next stdout line to the beginning of the line without proceeding to the next line.

---

Problem Formulation

Problem Definition: How will you overwrite the previous print/output to stdout in Python?

Example: Let’s say you have the following snippet, which prints the output as shown below:

import time for i in range(10): if i % 2 == 0: print(i, end="\r") time.sleep(2)

Output:

Challenge: What we want to do is instead of printing each output in a newline, we want to replace the previous output value and overwrite it with the new output value on the same line, as shown below.

Method 1: Using Carriage Return (‘\r’) Character

Approach: The simplest solution to the given problem is to use the carriage return (‘\r‘) character within your print statement to return the stdout to the start of the same print line without advancing to the next line. This leads to the next print statement overwriting the previous print statement.

Note: Read here to learn more about the carriage return escape character.

Code:

import time for i in range(10): if i % 2 == 0: print(i, end="\r") time.sleep(2) 

Output:

That’s easy! Isn’t it? Unfortunately, this approach is not completely foolproof. Let’s see what happens when we execute the following snippet:

import time li = ['start', 'Processing result']
for i in range(len(li)): print(li[i], end='\r') time.sleep(2)
print('Terminate')

Output:

print('Terminate') is unable to completely wipe out the previous output. Hence, the final output is erroneous.

Since we are executing each output generated by a print statement on top of the previous output, it is not possible to display an output properly on the same line if the following output has a shorter length than the output before.

FIX: To fix the above problem, instead of simply overwriting the output, we must clear the previous output before displaying the next output. This can be done with the help of the following ANSI sequence: “\x1b[2K“.

Code:

import time li = ['start', 'Processing result']
for i in range(len(li)): print(li[i], end='\r') time.sleep(2)
print(end='\x1b[2K') # ANSI sequence to clear the line where the cursor is located
print('Terminate')

Output:

Method 2: Clear Line and Print Using ANSI Escape Sequence

Approach: The idea here is to use an extra print statement instead of altering the end parameter of the print statement that is used to display the output. The extra print statement is used to move the cursor back to the previous line where the output was printed and then clear it out with the help of ANSI escape sequences.

Explanation:

• Print a line that ends with a new line initially.
• Just before printing the next output on the new line, perform a couple of operations with the help of ANSI escape sequences:
  1. Move the cursor up, i.e., to the previous output line using the escape sequence: ‘\033[1A‘.
  2. Clear the line using the escape sequence: ‘\x1b[2K‘
• Print the next output.

Code:

import time UP = '\033[1A'
CLEAR = '\x1b[2K'
for i in range(10): if i % 2 == 0: print(i) time.sleep(2) print(UP, end=CLEAR)

Output:

Discussion: Though this code might look a little more complex than the previous approach, it comes with a major advantage of the neatness of output. You don’t have to worry about the length of the previous output. Also, the cursor does not visually hinder the output being displayed.

Here’s a handy guide to escape sequences with respect to cursor movements:

ESCAPE SEQUENCE	CURSOR MOVEMENT
\033[<L>;<C>H	Positions the cursor. Puts the cursor at line L and column C.
\033[<N>A	Move the cursor up by N lines.
\033[<N>B	Move the cursor down by N lines.
\033[<N>C	Move the cursor forward by N columns.
\033[<N>D	Move the cursor backward by N columns.
\033[2J	Clear the screen, move to (0,0)
\033[K	Erase the end of line.

Method 3: Using “\b” Character

Another way to overwrite the previous output line is to use the backspace character(“\b“) and write to the standard output.

Code:

import time
import sys for i in range(10): if i % 2 == 0: sys.stdout.write(str(i)) time.sleep(1) sys.stdout.write('\b') sys.stdout.flush()

Output:

Caution: Ensure that you properly flush the buffer as done in the above snippet. Otherwise, you might see that only the last result is displayed at the end of the script.

Bonus Read Ahead

What is Carriage Return (\r) in Python?

Simply put, carriage return is an escape character just like \n. Carriage return is denoted as \r and it is basically used to shift the cursor to the beginning of a line or string instead of allowing it to move on to the next line.

Whenever you use the carriage return escape character ‘\r’, the content that comes after the \r will appear on top of your line and will keep replacing the characters of the previous string one by one until it occupies all the contents left after the \r in that string.

Example:

li = ['One', 'Two', 'Three']
for i in range(len(li)): print(li[i], end='\r') # OUTPUT-->Three

Conclusion

To sum things up, the easiest way to overwrite the previous print is to use the carriage return \r character within your print statement using the end parameter. To ensure that the previous output is completely erased before printing the new output, you can use the \x1b[2K ANSI escape sequence.

I hope this tutorial helped you. Here’s another interesting read that you may find useful: Python Print One Line List

---

• One of the most sought-after skills on Fiverr and Upwork is web scraping. Make no mistake: extracting data programmatically from websites is a critical life skill in today’s world that’s shaped by the web and remote work.
• So, do you want to master the art of web scraping using Python’s BeautifulSoup?
• If the answer is yes – this course will take you from beginner to expert in Web Scraping.

Before we learn about the money, let’s get this question out of the way:

What is Tableau?

Let’s have a look at the definition inspired by the official Tableau website:

Tableau is a visual data analytics platform focused on the business analytics use case that helps you use data to solve problems. It is great to visualize data (e.g., using dashboards) and perform complex data analytics tasks with relatively simple-to-use operations.

I recommend you watch the following 10-minute video at 1.5x speed to get a first intuition of the platform quickly:

Here’s a screenshot of the video that shows the data analysis process relevant for Tableau:

And here’s a screenshot of all the subproducts of the Tableau platform—all of them may be a potential specialization for a Tableau Developer:

So, Tableau helps you accomplish all five steps of business process and business intelligence:

1. Storing transactions
2. Analyse data
   • Data preparation
   • Data analysis
3. Sharing insights
4. Making decisions
5. Monitoring outcomes and results

What is a Tableau Developer?

A Tableau Developer analyzes data, develops software, and creates data visualizations using Tableau to make businesses more efficient and effective.

Tableau developers routinely generate Tableau dashboards, business intelligence (BI) reports, and data visualizations to improve decision making in a data-driven organization.

Now that you know about what it is, let’s have a look at what it earns next!

Annual Income of Tableau Developer (US)

Question: How much does a Tableau Developer in the US make per year?

The expected annual income of a Tableau Developer in the United States is between $71,807 and $114,559, with an average annual income of $93,564 and a median income of $93,460 per year.

This data is based on our meta-study of 8 salary aggregators sources such as Glassdoor, ZipRecruiter, and PayScale.

Source	Average Income
Glassdoor.com	$97,578
ZipRecruiter.com	$114,559
Talent.com	$110,000
Indeed.com	$105,000
PayScale.com	$75,401
Comparably.com	$71,807
Zippia.com	$89,343
Salary.com	$84,829

Note: This is the most comprehensive salary meta-study of tableau developer income in the world, to the best of my knowledge!

To learn more about the income potential and opportunity in the broad data space, have a look at our full guide on the Finxter blog:

• Data Engineer — Income and Opportunity
• Data Scientist — Income and Opportunity
• Machine Learning Engineer — Income and Opportunity

Let’s have a look at the hourly rate of tableau freelancers next!

Hourly Rate of Tableau Freelancers

Tableau Developers are well-paid on freelancing platforms such as Upwork or Fiverr.

• Related Article: What’s the best freelancing platform?

If you decide to go the route as a freelance Tableau Developer, you can expect to make between $40 and $80 per hour on Upwork (source). Assuming an annual workload of 2000 hours, you can expect to make between $80,000 and $160,000 per year.

Note: Do you want to create your own thriving coding business online? Feel free to check out our freelance developer course — the world’s #1 best-selling freelance developer course that specifically shows you how to succeed on Upwork and Fiverr!

Industry Demand

But is there enough demand? Let’s have a look at Google trends to find out how interest evolves over time (source):

Tableau is now a Salesforce company. So, here’s the general interest in Salesforce for your reference:

Both trends look optimistic—up into the right!

Learning Path, Skills, and Education Requirements

Do you want to become a Tableau Developer? Here’s a step-by-step learning path I’d propose to get started with Tableau :

• Step 1: Introduction to Computer Science (~40 hours)
• Step 2: Introduction to Machine Learning (~40 hours)
• Step 3: Introduction to Data Science (~40 hours)
• Step 4: Introduction to Tableau (~20 hours)
• Step 5: Introduction to Tableau (Free Video Courses) (~10 hours)
• Step 6: Official Tableau Training Program (~40 hours)

You can find many additional computer science courses on the Finxter Computer Science Academy (flatrate model).

But don’t wait too long to acquire practical experience!

Even if you have little skills, it’s best to get started as a freelance developer and learn as you work on real projects for clients — earning income as you learn and gaining motivation through real-world feedback.

Tip: An excellent start to turbo-charge your freelancing career (earning more in less time) is our Finxter Freelancer Course. The goal of the course is to pay for itself!

Related Video

You can find more job descriptions for coders, programmers, and computer scientists in our detailed overview guide:

• Get Your Job in IT [Overview]

Related Income of Professional Developers

The following statistic shows the self-reported income from 9,649 US-based professional developers (source).

The average annual income of professional developers in the US is between $70,000 and $177,500 for various programming languages.

Question: What is your current total compensation (salary, bonuses, and perks, before taxes and deductions)? Please enter a whole number in the box below, without any punctuation. If you are paid hourly, please estimate an equivalent weekly, monthly, or yearly salary. (source)

The following statistic compares the self-reported income from 46,693 professional programmers as conducted by StackOverflow.

The average annual income of professional developers worldwide (US and non-US) is between $33,000 and $95,000 for various programming languages.

Here’s a screenshot of a more detailed overview of each programming language considered in the report:

Here’s what different database professionals earn:

Here’s an overview of different cloud solutions experts:

Here’s what professionals in web frameworks earn:

There are many other interesting frameworks—that pay well!

Look at those tools:

Okay, but what do you need to do to get there? What are the skill requirements and qualifications to make you become a professional developer in the area you desire?

Let’s find out next!

General Qualifications of Professionals

StackOverflow performs an annual survey asking professionals, coders, developers, researchers, and engineers various questions about their background and job satisfaction on their website.

Interestingly, when aggregating the data of the developers’ educational background, a good three quarters have an academic background.

Here’s the question asked by StackOverflow (source):

Which of the following best describes the highest level of formal education that you’ve completed?

However, if you don’t have a formal degree, don’t fear! Many of the respondents with degrees don’t have a degree in their field—so it may not be of much value for their coding careers anyways.

Also, about one out of four don’t have a formal degree and still succeeds in their field! You certainly don’t need a degree if you’re committed to your own success!

Freelancing vs Employment Status

The percentage of freelance developers increases steadily. The fraction of freelance developers has already reached 11.21%!

This indicates that more and more work will be done in a more flexible work environment—and fewer and fewer companies and clients want to hire inflexible talent.

Here are the stats from the StackOverflow developer survey (source):

Do you want to become a professional freelance developer and earn some money on the side or as your primary source of income?

Resource: Check out our freelance developer course—it’s the best freelance developer course in the world with the highest student success rate in the industry!

Other Programming Languages Used by Professional Developers

The StackOverflow developer survey collected 58000 responses about the following question (source):

Which programming, scripting, and markup languages have you done extensive development work in over the past year, and which do you want to work in over the next year?

These are the languages you want to focus on when starting out as a coder:

And don’t worry—if you feel stuck or struggle with a nasty bug. We all go through it. Here’s what SO survey respondents and professional developers do when they’re stuck:

What do you do when you get stuck on a problem? Select all that apply. (source)

Related Tutorials

To get started with some of the fundamentals and industry concepts, feel free to check out these articles:

• Freelance Developer – How to Code From Home and Earn Six Figures [Industry Report]
• How to Become a Python Freelancer—and Earn $1,000 on the Side? [A Step-by-Step Tutorial]
• How Adam Earns $5000 per Month as a Python Freelancer on Upwork [Month 4]
• Tableau Wikipedia
• Learn Tableau Google

Where to Go From Here?

Enough theory. Let’s get some practice!

Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.

To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

You build high-value coding skills by working on practical coding projects!

Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?

If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!

References

[1] The figure was generated using the following code snippet:

import matplotlib.pyplot as plt
import numpy as np
import math data = [97578, 114559, 110000, 105000, 75401, 71807, 89343, 84829] labels = ['Glassdoor.com', 'ZipRecruiter.com', 'Talent.com', 'Indeed.com', 'PayScale.com', 'Comparably.com', 'Zippia.com', 'Salary.com'] median = np.median(data)
average = np.average(data)
print(median, average)
n = len(data) plt.plot(range(n), [median] * n, color='black', label='Median: $' + str(int(median)))
plt.plot(range(n), [average] * n, '--', color='red', label='Average: $' + str(int(average)))
plt.bar(range(len(data)), data)
plt.xticks(range(len(data)), labels, rotation='vertical', position = (0,0.45), color='white', weight='bold')
plt.ylabel('Average Income ($)')
plt.title('Tableau Developer Annual Income - by Finxter')
plt.legend()
plt.show()

Problem Formulation

Say, you want to find a regex pattern in a given string. You know the pattern exists in the string. You use the re.match(pattern, string) function to find the match object where the pattern matches in the string.

Problem: The Python regular expression pattern is not found in the string. The pattern doesn’t match anything, and, thus, the match object is None. How to fix this?

Here’s an example in which you’re searching for the pattern 'h[a-z]+' which should match the substring 'hello'.

But it doesn’t match!

import re my_string = 'hello world'
pattern = re.compile('h[a-z]+') match = re.match(pattern, my_string) if match: print('found!')
else: print('not found!')

Output:

not found!

Where is the bug? And how to fix it, so that the pattern matches the substring 'hello'?

Learn More: Improve your regex superpower by studying character classes used in the example pattern 'h[a-z]+' by visiting this tutorial on the Finxter blog.

Solution: Use re.search() instead of re.match()

A common reason why your Python regular expression pattern is not matching in a given string is that you mistakenly used re.match(pattern, string) instead of re.search(pattern, string) or re.findall(pattern, string). The former attempts to match the pattern at the beginning of the string, whereas the latter two functions attempt to match anywhere in the string.

Here’s a quick recap of the three regex functions:

• re.match(pattern, string) returns a match object if the pattern matches at the beginning of the string. The match object contains useful information such as the matching groups and the matching positions.
• re.search(pattern, string) matches the first occurrence of the pattern in the string and returns a match object.
• re.findall(pattern, string) scans string from left to right, searching for all non-overlapping matches of the pattern. It returns a list of strings in the matching order when scanning the string from left to right.

Thus, the following code uses re.search() to fix our problem:

import re my_string = 'hello world'
pattern = re.compile('h[a-z]+') match = re.search(pattern, my_string) if match: print('found!')
else: print('not found!')

Output:

found!

Finally, the pattern 'h[a-z]+' does match the string 'hello world'.

Note that you can also use the re.findall() function if you’re interested in just the string matches of your pattern (without match object). We’ll explain all of this — re.match(), re.search(), re.findall(), and match objects — in a moment but first, let’s have a look at the same example with re.findall():

import re my_string = 'hello world'
pattern = re.compile('h[a-z]+') match = re.findall(pattern, my_string) print(match)
# ['hello'] if match: print('found!')
else: print('not found!')

Output:

['hello']
found!

Understanding re.match()

The re.match(pattern, string) method returns a match object if the pattern matches at the beginning of the string. The match object contains useful information such as the matching groups and the matching positions. An optional argument flags allows you to customize the regex engine, for example to ignore capitalization.

Specification:

re.match(pattern, string, flags=0)

The re.match() method has up to three arguments.

• pattern: the regular expression pattern that you want to match.
• string: the string which you want to search for the pattern.
• flags (optional argument): a more advanced modifier that allows you to customize the behavior of the function. Want to know how to use those flags? Check out this detailed article on the Finxter blog.

We’ll explore them in more detail later.

Return Value:

The re.match() method returns a match object. You may ask (and rightly so):

Learn More: Understanding re.match() on the Finxter blog.

What’s a Match Object?

If a regular expression matches a part of your string, there’s a lot of useful information that comes with it: what’s the exact position of the match? Which regex groups were matched—and where?

The match object is a simple wrapper for this information. Some regex methods of the re package in Python—such as search()—automatically create a match object upon the first pattern match.

At this point, you don’t need to explore the match object in detail. Just know that we can access the start and end positions of the match in the string by calling the methods m.start() and m.end() on the match object m:

>>> m = re.search('h...o', 'hello world')
>>> m.start()
0
>>> m.end()
5
>>> 'hello world'[m.start():m.end()] 'hello'

In the first line, you create a match object m by using the re.search() method. The pattern 'h...o' matches in the string 'hello world' at start position 0.

You use the start and end position to access the substring that matches the pattern (using the popular Python technique of slicing).

---

Now that you understood the purpose of the match object, let’s have a look at the alternative to the re.match() function next!

Understanding re.search()

The re.search(pattern, string) method matches the first occurrence of the pattern in the string and returns a match object.

Specification:

re.search(pattern, string, flags=0)

The re.search() method has up to three arguments.

• pattern: the regular expression pattern that you want to match.
• string: the string which you want to search for the pattern.
• flags (optional argument): a more advanced modifier that allows you to customize the behavior of the function. Want to know how to use those flags? Check out this detailed article on the Finxter blog.

We’ll explore them in more detail later.

Return Value:

The re.search() method returns a match object. You may ask (and rightly so):

Learn More: Understanding re.search() on the Finxter blog.

Understanding re.findall()

The re.findall(pattern, string) method scans string from left to right, searching for all non-overlapping matches of the pattern. It returns a list of strings in the matching order when scanning the string from left to right.

Specification:

re.findall(pattern, string, flags=0)

The re.findall() method has up to three arguments.

• pattern: the regular expression pattern that you want to match.
• string: the string which you want to search for the pattern.
• flags (optional argument): a more advanced modifier that allows you to customize the behavior of the function. Want to know how to use those flags? Check out this detailed article on the Finxter blog.

We will have a look at each of them in more detail.

Return Value:

The re.findall() method returns a list of strings. Each string element is a matching substring of the string argument.

Learn More: Understanding re.findall() on the Finxter blog.

---

Now, this was a lot of theory! Let’s get some practice.

In my Python freelancer bootcamp, I’ll train you on how to create yourself a new success skill as a Python freelancer with the potential of earning six figures online.

The next recession is coming for sure, and you want to be able to create your own economy so that you can take care of your loved ones.

Check out my free “Python Freelancer” webinar now!

Join 20,000+ ambitious coders for free!

Problem Formulation and Solution Overview

In this article, you’ll learn how to print the contents of a List without surrounding brackets in Python.

To make it more fun, we have the following running scenario:

You are a student and need to memorize the first 10 elements in the Periodic Table. This data is currently saved in a List format. However, you would prefer it to display without brackets to remove any distractions.

Question: How would we write Python code to print a List without brackets?

We can accomplish this task by one of the following options:

• Method 1: Use print() and join()
• Method 2: Use print(*, sep)
• Method 3: Use slicing
• Method 4: Use join() and map()
• Bonus: Strip quote characters from Method 3.

---

Method 1: Use print() and join()

This method uses join() to access each element of the List passed. Then print() lets join() know which separator to concatenate (append) to each element. The result is a String.

periodic_els = ['H', 'He', 'Li', 'Be', 'B', 'C', 'N', 'O', 'F', 'Ne']
print(' / '.join(periodic_els))

This code declares a List of the first 10 element names of the Periodic Table and saves them to periodic_els.

Next, join() passes periodic_els as an argument and accesses each element, adding the appropriate separator character(s) as indicated in the print statement (' / ').

Finally, the output is sent to the terminal as a String data type.

Output

H / He / Li / Be / B / C / N / O / F / Ne

If we modified the print statement to have a comma (',') as a separator, the output would be as follows:

elements = ['H', 'He', 'Li', 'Be', 'B', 'C', 'N', 'O', 'F', 'Ne']
print(','.join(elements))

Output

H,He,Li,Be,B,C,N,O,F,Ne

 Note: We recommend trying different separating characters, including a space (' ') and an empty space ('') to see the output.

---

Method 2: Use print(*, sep)

This method uses print() passing *periodic_els, and a separator character (' ') as arguments to the same.

periodic_els = ['H', 'He', 'Li', 'Be', 'B', 'C', 'N', 'O', 'F', 'Ne']
print(*periodic_els, sep=' ')

 Note: Placing an asterisk (*) before the variable lets Python know to unpack/extract the variable, in this case, a List.

This code declares a List of the first 10 elements in the Periodic Table and saves them to periodic_els.

Inside the print() statement *periodic_els is passed as the first argument. This tells Python to unpack the List as indicated above.

Next, the sep argument is passed to print(). This will concatenate (append) character(s) between each element, in this case, a space (' ').

Finally, the output is sent to the terminal as a String data type.

Output

H He Li Be B C N O F Ne

---

Method 3: Use Slicing

This method uses slicing to remove the brackets. However, the single quote characters surrounding each element will remain.

periodic_els = ['H', 'He', 'Li', 'Be', 'B', 'C', 'N', 'O', 'F', 'Ne']
print(str(periodic_els)[1:-1])

This code declares a List of the first 10 elements in the Periodic Table and saves them to periodic_els.

Next, the print() statement converts the List to a string and then uses slicing to extract the output as indicated by [1:-1].

Finally, the output is sent to the terminal as a String data type.

Output

‘H’, ‘He’, ‘Li’, ‘Be’, ‘B’, ‘C’, ‘N’, ‘O’, ‘F’, ‘Ne’

---

Method 4: Use join() and map()

If periodic_els contained integers instead of strings, it would need to be converted to a String data type first. Then, join() and map() are used to output the contents without brackets.

periodic_els = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print(', '.join(map(str, periodic_els)))

This code declares a List of the first 10 elements in the Periodic Table and saves them to periodic_els.

Next, periodic_els is converted to a String, and the iterable map() object is accessed. Then, each element is evaluated, and a separator character
(‘, ‘) is placed between each element and concatenated.

Finally, the output is sent to the terminal as a String data type.

Output

1, 2, 3, 4, 5, 6, 7, 8, 9,10

---

Bonus: Strip quote characters from Method 3

This section expands on Method 3, where the resultant output contained quote characters (') surrounding each element. This can easily be removed by running the following code.

periodic_els = ['H', 'He', 'Li', 'Be', 'B', 'C', 'N', 'O', 'F', 'Ne']
print(str(periodic_els)[1:-1].replace("'", ""))

The only difference between this code from Method 3 and the code above is that replace("'", "") is appended to the end.

Output

H, He, Li, Be, B, C, N, O, F, Ne

---

Summary

These four (4) methods of how to print a List without Brackets should give you enough information to select the best one for your coding requirements.

Good Luck & Happy Coding!

---

What is Data Cleaning?

Data cleaning describes the process of turning messy data into clean datasets that can be used for research and data science purposes. For example, tidy data will be in a wide format: every column contains a variable, and every row contains one case. Also, data cleaning means getting rid of corrupt data with very little utility.

Related Tutorial: Data Preparation in Python

How to Clean Unstructured Data in Python?

Most data in the real world is messy and unstructured or semi-structured. Working in data science, most of your time will be spent on cleaning and structuring data.

In research, data from surveys or experiments is mostly already structured into Excel or CSV tables.

In companies, data can be stored in databases, Excel files, or distributed all over the company. This data can be in emails, documents, folders, images, and note apps. 

This study shows, that most companies are having problems handling unstructured or semi-structured data, and almost half of them don’t even know where their data is located.

Unstructured data includes videos, images, and text or speech messages. Unstructured data from the web is mainly acquired by web scraping.

Semi-structured data is data found in documents, emails, social media posts, and if acquired from the web, it can be in HTML, JSON, or any other web format. 

Is Web Scraping Legal?

Important: Even though web scraping is possible does not mean it is always legal!

If the data is publicly available and not copyrighted, it is mostly safe to scrape. But also pay attention to data privacy laws and do not scrape personal data.

Scraping data from social media websites, for example, is mostly illegal, as it is not publicly available without logging in and contains personal data.

There are also many services to get data via an API. To be safe, refer to this guide about what is legal when web scraping:

What is the Goal of Data Cleaning?

The goal of data cleaning and cleaning unstructured or semi-structured data is to create tidy data with which you can work. Tidy data will be in a wide format: every column contains a variable, and every row contains one case.

To demonstrate both perspectives, this article is divided into two parts:

• First, we will scrape, load, and wrangle some semi-structured data from the web.
• Second, we will clean this data. This second step is also valid for structured data as it is about finding missing, outliers and duplicates.

I recommend doing an exploratory data analysis before or during cleaning data to get a good feeling of the data you have. You can easily combine exploring and cleaning.

Easy Exploratory Data Analysis (EDA) in Python with Visualization

Python HTML scraping

First, we will scrape a table from Wikipedia of the largest cities in the world. Scraping from Wikipedia is legal because

1. The data is publicly available 
2. The data works under a creative commons deed, which means the content is free to copy, share and adapt

First, we load our packages:

• pandas as our go-to library for data wrangling and analysis,
• the standard library for HTTP requests, and
• BeautifulSoup for easily reading HTML data.

import pandas as pd # pandas for data wrangling, cleaning, and analysis
import requests # for http requests
from bs4 import BeautifulSoup # html reading

The wiki page that holds the table we are looking for can be found here:

• https://en.wikipedia.org/wiki/List_of_largest_cities

We pass this URL into requests. The table in the HTML page is within <table class> , and the tables in wiki pages are called wiki tables. We can check this by looking at the HTML page in our browser or the HTML text file later to confirm we pulled the right table. 

With requests.get(url).text, we pull the HTML from the page. 

url='https://en.wikipedia.org/wiki/List_of_largest_cities'
url_response=requests.get(url).text

BeautifulSoup will pull the data table from the HTML file and save us time. We will pass the url_response from our request into the html.parser. With soup.find() we can tell it to look exactly for the wikitable. The output also tells us the name of the table.

soup = BeautifulSoup(url_response.text, 'html.parser')
html_listofcities=soup.find('table',{'class':'wikitable'})
html_listofcities

Output: 

Then we use pandas to read the HTML file and turn it into a pandas data frame, just like we would load any other data into Python.

df=pd.read_html(str(listofcities))
cities=pd.DataFrame(df[0])
print(cities.head())

Output:

The data looks messy. Now we get to clean!

Python HTML Cleaning

Let us inspect our newfound data:

Cities

Output:

I will drop the columns of the metropolitan area and the urban area because I am just interested in the population of the actual city. This can be done in several ways.

Here are two:

cities.drop(cities.columns[[7, 8, 9, 10, 11, 12]], axis=1, inplace=True)
# or
cities= cities.drop(cities.iloc[:,7:], axis = 1)

Then we will drop the first header column as it does not contain any useful information and rename the remaining header column.

# Dropping the first index column
cities.columns = cities.columns.droplevel(0) 

Output:

# rename the header column
cities.columns = ['city', 'country', 'UN_2018_population_estimates', 'city_definition', 'city_population', 'city_area_km2', ' city_density/km2']

Output:

The heading looks clean. Now we explore the dataset to find information to clean.

With df.info() and df.describe() we get a quick overview of the data we scraped. 

cities.info()

Output:

cities.describe()

Output:

It is immediately clear that the city_density/km2 is not a float even though it is supposed to be numerical.

Inspecting the data frame, you might have already noticed that the columns contain numbers following numbers in brackets, like [12]. This turns this data into an object, so we will have to get rid of this.

There are different ways to remove characters from a string in Python. We could just remove the last three characters of each string.

However, this would not work if some of our data points do not have the brackets at the end or more than that. So we’ll use the slicing method str.partition() to cut the brackets from our numbers. 

First, we make sure our object type is a string that we can work string operations on. Then we apply the str.partition() method and advise the function to cut off at the first bracket [.

cities[' city_density/km2'] = cities['city_density/km2'].astype('string')
city_density_str = cities['city_density/km2'].str.partition('[')
print(city_density_str)

Output:

What we want is the first column, so we pick this one [0] and write it back on a variable. 

city_density_str_col_1 = city_density_str[0]
print(city_density_str_col_1)

Output:

The commas in the variable will prevent us from converting the string into a float, so we’ll remove the comma with str.replace() before turning the string to a float with s.astype('float') and assigning it back to our data frame.

city_density_str_col_1 = city_density_str_col_1.str.replace(',','')
cities['city_density/km2'] = city_density_str_col_1.astype('float')
print(cities['city_density/km2'])

Output:

The variable now shows up when we look at df.describe() and we’ll want the results rounded for better readability:

cities.describe().round(1)

Output:

Cleaning Structured Data in Python

Following the cleaning of the scraped data we can now use it like a structured data frame with data we collected or downloaded.

This also can be cleaned of missing data, outliers and duplicates but does not always need data wrangling. However, with a data frame with many strings the cleaning process also often involves a lot of string manipulation.

Important note:

If you want to apply machine learning algorithms to your data, do split your dataset before feature engineering and data transformation as this can create data leakage!

Dropping duplicates is easy: 

cities = cities.drop_duplicates()

There didn’t seem to be duplicates in our df, as the size remained the same.

When dealing with missing values, we must decide how to handle them based on our data. 

We can either

• Drop missing values
• Replace or impute the values
• Leave missing values in the dataset 
• Transform the information that they’re missing into a new variable

First, we inspect our missing data. The function df.isnull() is a boolean function, that tells us for the whole data frame if data is missing or not.

We can sum it up to determine, how many values are missing in each column.

cities.isnull().sum()

Output:

We can drop rows with missing values completely.

This will cause us to lose useful information in other columns. But as the first row is completely empty anyway, we can drop this one.

The df.dropna() function has useful features that help us pick what missing data we want to remove. So, I just want to remove the one row, or all of them if there are more, with all missing values.

cities = cities.dropna(how='all')

This will look like this:

What is left are the missing values for 8 cities for population, area and density. We will replace those. 

Of course, you can look up the data on Wikipedia and reinsert them. For the sake of the exercise and because most of the time it is not possible to look up missing data, we will not do this.

The dataset now contains the data of the 73 biggest cities in the world, using the average of these to impute the missing values in the other 8 is the only and closest guess we have. This does not create much more information but keeps us from losing other information from these 8 cities.

The alternative option would be to drop those 8 cities completely.

So, we’ll replace the missing values in the area column with the average area size of all the other cities. First, we create the mean of the city area sizes, then we fill the missing values in the column with this value.

Pandas has the right function for this: df.fillna()

city_area_mean = cities['city_area_km2'].mean()
cities['city_area_km2'] = cities['city_area_km2'].fillna(value=city_area_mean)
cities.isnull().sum()

Output:

The output shows that we now have replaced and eliminated several missing values.

For the population, we luckily have another column that shows the population estimates of the UN in 2018 for each city.

So, we can use these to impute the missing population data, as it is as close as we can get to replacing them accurately. 

cities['city_population'] = cities['city_population'].fillna(cities['UN_2018_population_estimates'])
cities['city_population'].describe().round(1)

Output:

Now we still have missing values in the city density column. This one we can calculate now by dividing the population by the area.

So, we create a new variable to calculate the density with our new imputed data. Then we fill the missing values with this calculated density.

city_density = cities['city_population']/cities['city_area_km2']
cities['city_density/km2'] = cities['city_density/km2'].fillna(value=city_density)
cities['city_density/km2'].describe().round(1)

Output:

We can check back our missing values and the description of our dataset.

cities.isnull().sum()

cities.describe().round(1)

There is still one value missing in our city definition. Let’s have a look at these categories. 

cities['city_definition'].value_counts()

Output:

As we don’t know if the missing city is a municipality or a capital, we could just replace the missing value with the generic description of “city”, as we know they all are cities.

If you’d want to calculate the differences between these categories, it would be useful to categorize and merge these single entries into bigger categories.

For now, we will just replace the missing value with “city”, as I am more interested in the size of the cities than the category.

cities['city_definition'] = cities['city_definition'].fillna('City')
cities.isnull().sum()

Great! We got rid of all the missing values.

Info: For many statistical operations, missing values will be dropped by default and don’t create a problem. For machine learning algorithms missing values must be removed before modelling.

We can also create dummy variables (information is missing/ not missing) as the fact that the data is missing might be useful information. This way, the fact that they’re missing can be included in the data analysis process. 

Visualization

Now we visualize our data and check for outliers with a seaborn scatterplot.

import seaborn as sns
sns.scatterplot(data=cities, x="city_population", y="city_area_km2", size="city_population")

Output:

The city in the right top corner is clearly an outlier, but not one we would want to remove or equalize as it is not a measurement error. It is just the biggest city (or metropolitan area) in the world!

Let’s find out which one it is with df.sort_values(), using ascending=False to sort the city population from high to low.

cities.sort_values('city_population', ascending=False)

Output:

The biggest city on earth by size and city population is the municipality Chongqing in China with over 32 million inhabitants!

To visualize our cities more beautifully, we can use a sns.relplot() with color and different sizing.

sns.relplot(x="city_population", y="city_area_km2", hue="country", size="city_population", sizes=(40, 400), alpha=.5, palette="muted", height=6, data=cities)

Output:

Now keep on cleaning!

Problem Formulation and Solution Overview

In this article, you’ll learn how to check a List for Duplicates in Python.

To make it more fun, we have the following running scenario:

The Finxter Academy has given you an extensive list of usernames. Somewhere along the line, duplicate entries were added. They need you to check if their List contains duplicates. For testing purposes, a small sampling of this List is used.

Question: How would we write Python code to check a List for duplicate elements?

We can accomplish this task by one of the following options:

• Method 1: Use set() and List to return a Duplicate-Free List
• Method 2: Use set(), For loop and List to return a List of Duplicates found.
• Method 3: Use a For loop to return Duplicates and Counts
• Method 4: Use any() to check for Duplicates and return a Boolean
• Method 5: Use List Comprehension to return a List of all Duplicates

---

Method 1: Use set() and List to return a Duplicate-Free List

This method uses set()which removes any duplicate values (set(users)) to produce a Duplicate-Free set(). This set is then converted to a List (list(set(users))).

users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] dup_free = list(set(users))
print(dup_free)

This code declares a small sampling of Finxter usernames and saves them to users.

Next, set() is called and users is passed as an argument to the same. Then, the new set is converted to a List and saved to dup_free.

If dup_free was output to the terminal before converting to a List, the result would be a set(), which is not subscriptable. Meaning the elements are inaccessible in this format.

Output

{'csealker', 'cdriver', 'shoeguy', 'ollie3', 'kyliek', 'stewieboy', 'AmyP'}

Note: Any attempt to access an element from a set will result in a not subscriptable error.

In this example, the set() was converted to a List, and displays a List of Duplicate-Free values.

Output

['csealker', 'cdriver', 'shoeguy', 'ollie3', 'kyliek', 'stewieboy', 'AmyP']

Note: An empty set will result if no argument is passed.

---

Method 2: Use set(), For loop, and List to return a List of Duplicates Found

This method uses set(), and a For loop to check for and return any Duplicates found (set(x for x in users if ((x in tmp) or tmp.add(x)))) to dups. The set() is then converted to a List (print(list(dups))).

Here’s an example:

users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] tmp = set()
dups = set(x for x in users if (x in tmp or tmp.add(x)))
print(list(dups))

This code declares a small sampling of Finxter usernames and saves them to users.

Next, a new empty set, tmp is declared. A For loop is then instantiated to check each element in users for duplicates. If a duplicate is found, it is appended to tmp. The results save to dups as a set().

Output

In this example, the set() was converted to a List and displays a List of Duplicates values found in the original List, users.

['kyliek', 'ollie3', 'shoeguy']

---

Method 3: Use a For loop to return Duplicates and Counts

This method uses a For loop to navigate through and check each element of users while keeping track of all usernames and the number of times they appear. A Dictionary of Duplicates, including the Usernames and Counts returns.

Here’s an example:

count = {}
dup_count = {}
for i in users: if i not in count: count[i] = 1 else: count[i] += 1 dup_count[i] = count[i]
print(dup_count)

This code declares two (2) empty sets, count and dup_count respectively.

A For loop is instantiated to loop through each element of users and does the following:

• If the element i is not in count, then the count element (count[i]=1) is set to one (1).
• If element i is found in count, it falls to else where one (1) is added to count (count[i]+=1) and then added to dup_count (dup_count[i]=count[i])

This code repeats until the end of users has been reached.

At this point, a Dictionary containing the Duplicates, and the number of times they appear displays.

Output

{'ollie3': 2, 'shoeguy': 2, 'kyliek': 2}

---

Method 4: Use Any to Check for Duplicate Values

This example uses any(), and passes the iterable users to iterate and locate Duplicates. If found, True returns. Otherwise, False returns. Best used on small Lists.

users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] dups = any(users.count(x) > 1 for x in users)
print(dups)

This code declares a small sampling of Finxter usernames and saves them to users.

Next, any() is called and loops through each element of users checking to see if the element is a duplicate. If found, True is assigned. Otherwise, False is assigned. The result saves to dups and the output displays as follows:

Output

True

---

Method 5: Use List Comprehension to return a List of all Duplicates

This method uses List Comprehension to loop through users, checking for duplicates. If found, the Duplicates are appended to dups.

Here’s an example:

users = ['AmyP', 'ollie3', 'shoeguy', 'kyliek', 'ollie3', 'stewieboy', 'csealker', 'shoeguy', 'cdriver', 'kyliek'] dups = [x for x in users if users.count(x) >= 2]
print(dups)

This code declares a small sampling of Finxter usernames and saves them to users.

Next, List Comprehension extracts and displays duplicate usernames and save them to a List. The duplicate values are output to the terminal

Output

['ollie3', 'shoeguy', 'kyliek', 'ollie3', 'shoeguy', 'kyliek']

---

Summary

These five (5) methods of checking a List for Duplicates should give you enough information to select the best one for your coding requirements.

Good Luck & Happy Coding!

---