Summary: You can split a string using an empty separator using – (i) list constructor (ii) map+lambda (iii) regex (iv) list comprehension
Minimal Example:
text = '12345' # Using list()
print(list(text)) # Using map+lambda
print(list(map(lambda c: c, text))) # Using list comprehension
print([x for x in text]) # Using regex
import re
# Approach 1
print([x for x in re.split('', text) if x != ''])
# Approach 2
print(re.findall('.', text))
Problem Formulation
Problem: How to split a string using an empty string as a separator?
Example: Consider the following snippet –
a = 'abcd'
print(a.split(''))
Output:
Traceback (most recent call last): File "C:\Users\SHUBHAM SAYON\PycharmProjects\Finxter\Blogs\Finxter.py", line 2, in <module> a.split('')
ValueError: empty separator
Expected Output:
['a', 'b', 'c', 'd']
So, this essentially means that when you try to split a string by using an empty string as the separator, you will get a ValueError. Thus, your task is to find out how to eliminate this error and split the string in a way such that each character of the string is separately stored as an item in a list.
Now that we have a clear picture of the problem let us dive into the solutions to solve the problem.
Approach: Use the list() constructor and pass the given string as an argument within it as the input, which will split the string into separate characters.
Note: list() creates a new list object that contains items obtained by iterating over the input iterable. Since a string is an iterable formed by combining a group of characters, hence, iterating over it using the list constructor yields a single character at each iteration which represents individual items in the newly formed list.
Approach: Use the map() to execute a certain lambda function on the given string. All you need to do is to create a lambda function that simply returns the character passed to it as the input to the map object. That’s it! However, the map method will return a map object, so you must convert it to a list using the list() function.
The re.findall(pattern, string) method scans string from left to right, searching for all non-overlapping matches of the pattern. It returns a list of strings in the matching order when scanning the string from left to right.
Approach: Use the regular expression re.findall('.',a) that finds all characters in the given string ‘a‘ and stires them in a list as individual items.
Code:
import re
a = 'abcd'
print(re.findall('.',a)) # ['a', 'b', 'c', 'd']
Alternatively, you can also use the split method of the regex library in a list comprehension which returns each character of the string and eliminates empty strings.
Code:
import re
a = 'abcd'
print([x for x in re.split('',a) if x!='']) # ['a', 'b', 'c', 'd']
Do you want to master the regex superpower? Check out my new book The Smartest Way to Learn Regular Expressions in Python with the innovative 3-step approach for active learning: (1) study a book chapter, (2) solve a code puzzle, and (3) watch an educational chapter video.
Conclusion
Hurrah! We have successfully solved the given problem using as many as four (five, to be honest) different ways. I hope this article helped you and answered your queries. Please subscribe and stay tuned for more interesting articles and solutions in the future.
Happy coding!
Regex Humor
Wait, forgot to escape a space. Wheeeeee[taptaptap]eeeeee. (source)
Say, you try to import label_map_util from the utils module when running TensorFlow’s object_detection API. You get the following error message:
>>> from utils import label_map_util
Traceback (most recent call last): File "<pyshell#3>", line 1, in <module> from utils import label_map_util
ModuleNotFoundError: No module named 'utils'
Question: How to fix the ModuleNotFoundError: No module named 'utils'?
Solution Idea 1: Fix the Import Statement
The most common source of the error is that you use the expression from utils import <something> but Python doesn’t find the utils module. You can fix this by replacing the import statement with the corrected from object_detection.utils import <something>.
For example, do not use these import statements:
from utils import label_map_util
from utils import visualization_utils as vis_util
Instead, use these import statements:
from object_detection.utils import label_map_util
from object_detection.utils import visualization_utils as vis_util
Everything remains the same except the bolded text.
This, of course, assumes that Python can resolve the object_detection API. You can follow the installation recommendations here, or if you already have TensorFlow installed, check out the Object Detection API installation tips here.
Solution Idea 2: Modify System Path
Another idea to solve this issue is to append the path of the TensorFlow Object Detection API folder to the system paths so your script can find it easily.
To do this, import the sys library and run sys.path.append(my_path) on the path to the object_detection folder that may reside in /home/.../tensorflow/models/research/object_detection, depending on your environment.
I don’t recommend using this approach but I still want to share it with you for comprehensibility. Try copying the utils folder from models/research/object_detection in the same directory as the Python file requiring utils.
Solution Idea 4: Import Module from Another Folder (Utils)
This is a better variant of the previous approach: use our in-depth guide to figure out a way to import the utils module correctly, even though it may reside on another path. This should usually do the trick.
Resources: You can find more about this issue here, here, and here. Among other sources, these were also the ones that inspired the solutions provided in this tutorial.
Thanks for reading this—feel free to learn more about the benefits of a TensorFlow developer (we need to keep you motivated so you persist through the painful debugging process you’re currently in).
Summary: Split the string using split and then use len to count the results.
Minimal Example
print("Result Count: ", len('one,two,three'.split(',')))
# Result Count: 3
Problem Formulation
Problem: Given a string. How will you split the string and find the number of split strings? Can you store the split strings into different variables?
Example
# Given String
text = '366NX-BQ62X-PQT9G-GPX4H-VT7TX'
# Expected Output:
Number of split strings: 5
key_1 = 366NX
key_2 = BQ62X
key_3 = PQT9G
key_4 = GPX4H
key_5 = VT7TX
In the above problem, the delimiter used to split the string is “-“. After splitting five substrings can be extracted. Therefore, you need five variables to store the five substrings. Can you solve it?
Solution
Splitting the string and counting the number of results is a cakewalk. All you have to do is split the string using the split() function and then use the len method upon the resultant list returned by the split method to get the number of split strings present.
Code:
text = '366NX-BQ62X-PQT9G-GPX4H-VT7TX'
# splitting the string using - as separator
res = text.split('-')
# length of split string list
x = len(res)
print("Number of split strings: ", x)
Approach 1
The idea here is to find the length of the results and then use this length to create another list containing all the variable names as items within it. This can be done with a simple for loop.
Now, you have two lists. One that stores the split strings and another that stores the variable names that will store the split strings.
So, you can create a dictionary out of the two lists such that the keys in this dictionary will be the items of the list containing the variable names and the values in this dictionary will be the items of the list containing the split strings. Read: How to Convert Two Lists Into A Dictionary
Code:
text = '366NX-BQ62X-PQT9G-GPX4H-VT7TX'
# splitting the string using - as separator
res = text.split('-')
# length of split string list
x = len(res) # Naming and storing variables and values
name = []
for i in range(1, x+1): name.append('key_'+str(i)) d = dict(zip(name, res))
for key, value in d.items(): print(key, "=", value)
Approach 2
Almost all modules have a special attribute known as __dict__ which is a dictionary containing the module’s symbol table. It is essentially a dictionary or a mapping object used to store an object’s (writable) attributes.
So, you can create a class and then go ahead create an instance of this class which can be used to set different attributes. Once you split the given string and also create the list containing the variable names (as done in the previous solution), you can go ahead and zip the two lists and use the setattr() method to assign the variable and their values which will serve as the attributes of the previously created class object. Once, you have set the attributes (i.e. the variable names and their values) and attached them to the object, you can access them using the built-in __dict__ as object_name.__dict__
Code:
text = '366NX-BQ62X-PQT9G-GPX4H-VT7TX'
# splitting the string using - as separator
res = text.split('-')
# length of split string list
x = len(res)
print("Number of split strings: ", x) # variable creation and value assignment
name = []
for i in range(1, x + 1): name.append('key_' + str(i)) class Record(): pass r = Record() for name, value in zip(name, res): setattr(r, name, value)
print(r.__dict__) for key, value in r.__dict__.items(): print(key, "=", value)
Approach 3
Caution: This solution is not recommended unless this is the only option left. I have mentioned this just because it solves the purpose. However, it is certainly not the best way to approach the given problem.
Code:
text = '366NX-BQ62X-PQT9G-GPX4H-VT7TX'
# splitting the string using - as separator
res = text.split('-')
# length of split string list
x = len(res)
print("Number of split strings: ", x)
name = []
for i in range(1, x + 1): name.append('key_' + str(i)) for idx, value in enumerate(res): globals()["key_" + str(idx + 1)] = value
print(globals())
x = 0
for i in reversed(globals()): print(i, "=", globals()[i]) x = x+1 if x == 5: break
Explanation: globals() function returns a dictionary containing all the variables in the global scope with the variable names as the key and the value assigned to the variable will be the value in the dictionary. You can reference this dictionary and add new variables by string name (globals()['a'] = 'b' sets variable a equal to "b"), however this is generally a terrible thing to do.
Since global returns a dictionary containing all the variables in the global scope, a workaround to get only the variables we assigned is to extract the last “N” key-value pairs from this dictionary where “N” is the length of the split string list.
Conclusion
I hope the solutions mentioned in this tutorial have helped you. Please stay tuned and subscribe for more interesting reads and solutions in the future. Happy coding!
Question: Given a Python list of hexadecimal strings such as ['ff', 'ef', '0f', '0a', '93']. How to convert it to a list of integers in Python such as [255, 239, 15, 10, 147]?
Easiest Answer
The easiest way to convert a list of hex strings to a list of integers in Python is the list comprehension statement [int(x, 16) for x in my_list] that applies the built-in function int() to convert each hex string to an integer using the hexadecimal base 16, and repeats this for each hex string x in the original list.
Here’s a minimal example:
my_list = ['ff', 'ef', '0f', '0a', '93']
my_ints = [int(x, 16) for x in my_list] print(my_ints)
# [255, 239, 15, 10, 147]
The list comprehension statement applies the expression int(x, 16) to each element x in the list my_list and puts the result of this expression in the newly-created list.
The int(x, 16) expression converts a hex string to an integer using the hexadecimal base argument 16. A semantically identical way to write this would be int(x, base=16).
In fact, there are many more ways to convert a hex string to an integer—each of them could be used in the expression part of the list comprehension statement.
However, I’ll show you one completely different approach to solving this problem without listing each and every combination of possible solutions.
For Loop with List Append
You can create an empty list and add one hex integer at a time in the loop body after converting it from the hex string x using the eval('0x' + x) function call. This first creates a hexadecimal string with '0x' prefix using string concatenation and then lets Python evaluate the string as if it was real code and not a string.
Here’s an example:
my_list = ['ff', 'ef', '0f', '0a', '93'] my_ints = []
for x in my_list: my_ints.append(eval('0x' + x)) print(my_ints)
# [255, 239, 15, 10, 147]
You use the fact that Python automatically converts a hex value of the form 0xff to an integer 255:
>>> 0xff
255
>>> 0xfe
254
>>> 0x0f
15
You may want to check out my in-depth guide on this important function for our solution:
While working on the second box in the series, Looking Glass, I stumbled upon a bash script written by Tay1or, another user on TryHackMe.
The opening challenge involves finding the correct port which hides an encrypted poem, Jabberwocky by Lewis Caroll.
Using a script here is a more efficient solution because it is quite time-consuming to manually attempt connecting to different ssh ports over and over until the correct port can be found.
The box also resets the mystery port after each login, so unless you solve the box on your first attempt, the script will come in handy multiple times.
Bash Script
Here is Tay1or’s bash script with a few slight modifications in bold to make it run on my machine:
#!/usr/bin/bash low=9000
high=13000 while true
do mid=$(echo "($high+$low)/2" | bc) echo -n "Low: $low, High: $high, Trying port: $mid – " msg=$(ssh -o "HostKeyAlgorithms=+ssh-rsa" -p $mid $targetIP | tr -d '\r') echo "$msg" if [[ "$msg" == "Lower" ]] then low=$mid elif [[ "$msg" == "Higher" ]] then high=$mid fi
done
I’m still new to bash scripting, but because I already understand the context of the problem being faced, I can more or less guess what the script is doing.
At the top, under the shebang line, it first sets low and high values for the ports to be searched. Then we see a while true loop.
The first command in the loop calculates the midpoint between the low and the high port values in the given range.
The echo command prints the low/high/and midpoint port that is currently being tested.
Then we have if/elif commands to respond appropriately to the output of the $msg to set the mid to either the lower or higher range variables. By resetting the range after each attempted connection, the search will take a minimal amount of time by eliminating the largest number of ports possible on each attempt.
When the output msg is neither “Higher” or “Lower” it will end the loop because we will have hit our secret encrypted message on the correct port.
Conversion into a Python script
I started wondering how it might be possible to translate the bash script to a Python script and decided to try my hand at converting the functionality of the code.
I’m more comfortable scripting in Python, and I think it will probably come in handy later in future challenges to be able to quickly write up a script during CTF challenges to save time.
The inputs of the code are the targetIP and high and low values of the target SSH port range.
Outputs are the response from the targetIP on each attempted connection until the secret port is found. Once the secret port is found, the program will reiterate that you have found the port.
I posted the final version of the python script here on GitHub. For your convenience, I’ll include it here too:
#!/usr/bin/env python3
# These sites were used as references: https://stackabuse.com/executing-shell-commands-wi>
# https://stackoverflow.com/questions/4760215/running-shell-command-and-capturing-the-> #set up initial conditions for the target port search
import subprocess
low_port=9000
high_port=13790
targetIP = "10.10.252.52"
print(targetIP)
#initialize loop_key variable:
loop_key="higher" while loop_key=="Higher" or "Lower": print('low = ' + str(low_port) + ', high = ' + str(high_port))
#a good place to use floor division to cut off the extra digit mid_port=(high_port+low_port)//2 print('Trying port ' + str(mid_port)) #attempt to connect to the mid port result = subprocess.run(['ssh', 'root@' + str(targetIP), '-oHostKeyAlgorithms=+ssh-rsa', '-p', str(mid_port)], stdout=subprocess.PIPE) # prep the decoded output variable msg = result.stdout decoded_msg = msg.decode('utf-8') # print result of attempted ssh connection print(decoded_msg) if "Higher" in decoded_msg: #print("yes I see the words Higher") high_port=mid_port print(high_port) loop_key="Higher" elif "Lower" in decoded_msg: low_port=mid_port print(low_port) loop_key="Lower" else: print("You found the secret port - " + str(mid_port)) exit()
Quick Fix: Python raises the ModuleNotFoundError: No module named 'git' when you haven’t installed GitPython explicitly with pip install GitPython or pip3 install GitPython (Python 3). Or you may have different Python versions on your computer, and GitPython is not installed for the particular version you’re using.
You’ll learn how to fix this error below after diving into the concrete problem first.
Problem Formulation
You’ve just learned about the awesome capabilities of the GitPython library, and you want to try it out, so you start your code with the following statement:
import git
or even something like:
from git import Repo
This is supposed to import the git library into your (virtual) environment. However, it only throws the following ModuleNotFoundError: No module named 'git':
>>> import git
Traceback (most recent call last): File "<pyshell#6>", line 1, in <module> import git
ModuleNotFoundError: No module named 'git'
Solution Idea 1: Install Library GitPython
The most likely reason is that Python doesn’t provide GitPython in its standard library. You need to install it first!
To fix this error, you can run the following command in your Windows shell:
$ pip install GitPython
This simple command installs GitPython in your virtual environment on Windows, Linux, and MacOS. Uppercase or lowercase library name doesn’t matter.
This assumes that your pip version is updated. If it isn’t, use the following two commands in your terminal, command line, or shell (there’s no harm in doing it anyways):
Note: Don’t copy and paste the $ symbol. This is just to illustrate that you run it in your shell/terminal/command line.
Also, try the following variant if you run Python 3 on your computer:
pip3 install GitPython
Solution Idea 2: Fix the Path
The error might persist even after you have installed the GitPython library. This likely happens because pip is installed but doesn’t reside in the path you can use. Although pip may be installed on your system the script is unable to locate it. Therefore, it is unable to install the library using pip in the correct path.
To fix the problem with the path in Windows follow the steps given next.
Step 1: Open the folder where you installed Python by opening the command prompt and typing where python
Step 2: Once you have opened the Python folder, browse and open the Scripts folder and copy its location. Also verify that the folder contains the pip file.
Step 3: Now open the Scripts directory in the command prompt using the cd command and the location that you copied previously.
Step 4: Now install the library using pip install GitPython command. Here’s an analogous example:
After having followed the above steps, execute our script once again. And you should get the desired output.
Other Solution Ideas
The ModuleNotFoundError may appear due to relative imports. You can learn everything about relative imports and how to create your own module in this article.
You may have mixed up Python and pip versions on your machine. In this case, to install GitPython for Python 3, you may want to try python3 -m pip install GitPython or even pip3 install GitPython instead of pip install GitPython
If you face this issue server-side, you may want to try the command pip install – user GitPython
If you’re using Ubuntu, you may want to try this command: sudo apt install GitPython
You can also check out this article to learn more about possible problems that may lead to an error when importing a library.
Understanding the “import” Statement
import git
In Python, the import statement serves two main purposes:
Search the module by its name, load it, and initialize it.
Define a name in the local namespace within the scope of the import statement. This local name is then used to reference the accessed module throughout the code.
What’s the Difference Between ImportError and ModuleNotFoundError?
What’s the difference between ImportError and ModuleNotFoundError?
Python defines an error hierarchy, so some error classes inherit from other error classes. In our case, the ModuleNotFoundError is a subclass of the ImportError class.
You can see this in this screenshot from the docs:
You can also check this relationship using the issubclass() built-in function:
Specifically, Python raises the ModuleNotFoundError if the module (e.g., git) cannot be found. If it can be found, there may be a problem loading the module or some specific files within the module. In those cases, Python would raise an ImportError.
If an import statement cannot import a module, it raises an ImportError. This may occur because of a faulty installation or an invalid path. In Python 3.6 or newer, this will usually raise a ModuleNotFoundError.
Related Videos
The following video shows you how to resolve the ImportError:
How to Fix “ModuleNotFoundError: No module named ‘git’” in PyCharm
If you create a new Python project in PyCharm and try to import the GitPython library, it’ll raise the following error message:
Traceback (most recent call last): File "C:/Users/.../main.py", line 1, in <module> import git
ModuleNotFoundError: No module named 'git' Process finished with exit code 1
The reason is that each PyCharm project, per default, creates a virtual environment in which you can install custom Python modules. But the virtual environment is initially empty—even if you’ve already installed GitPython on your computer!
Here’s a screenshot exemplifying this for the pandas library. It’ll look similar for GitPython.
The fix is simple: Use the PyCharm installation tooltips to install Pandas in your virtual environment—two clicks and you’re good to go!
First, right-click on the pandas text in your editor:
Second, click “Show Context Actions” in your context menu. In the new menu that arises, click “Install Pandas” and wait for PyCharm to finish the installation.
The code will run after your installation completes successfully.
As an alternative, you can also open the Terminal tool at the bottom and type:
In my previous Wonderland walkthrough blog post, I highlighted an example of exploiting the ‘random’ module to switch users without knowing their password.
In this post, I’ll guide you through the setup and execution of the exploit. You can also watch the accompanying video tutorial here:
What is Python Library Hijacking?
When a user has permission to run a file as another user it is possible to create a spoof file that Python will load instead of the originally intended module or library. The necessary conditions for Python library hijacking are:
The user must have sudo permissions to run a Python file .py as another user
The Python path must be set to look first in the folder where the spoof file is stored
Setup
In order to re-create this vulnerability, I had to learn how to set up the above conditions for the exploit.
On my home network, I have a Raspberry Pi 3b running DietPi operating system. Originally I set this up to run Pi-hole to filter ads out from my home network.
In order to set up the permissions to run a file as another user I edited the sudoers file with visudo.
Visudo is a special editor specifically for editing the sudoers file. It only allows one user to edit the file at a time, and also checks user edits for correct syntax. I created a file called ‘checkmypermissions.py’ and granted sudo permissions to vulnerableuser to run it as user ben.
To do this I used the command ‘sudo visudo’ to edit sudoers file, and then I added the second line for vulnerable user:
# User privilege specification
root ALL=(ALL:ALL) ALL
vulnerableuser ALL=(ben:1001) /usr/bin/python3 /home/vulnerableuser/checkmypermissions.py
The nice thing about visudo is that it checks your formatting to make sure that there are not any errors, and it will even suggest changes to help you format the permissions correctly.
This functionality helped me save time getting the correct spacing and punctuation on the new sudoers line.
Running the Exploit
Once the permissions were set up I ssh’d into vulnerableuser@<raspberry pi IP>. Running the ‘sudo -l’ command showed me the granular sudo permissions.
The line above (ben : 1001) /usr/bin/python3 /home/vulnerableuser/checkmypermissions.py shows that as vulnerableuser I can execute the checkmypermissions.py file as the user Ben.
All that is left to do is to check the Python PATH to make sure that it checks first in the current directory, and then create a python file named numpy.py with code to spawn a shell. One way to check the Python PATH is:
Python
import sys
sys.path
In the example below, we can see that the python PATH is already set to search in the current working directory ('').
Next we create the numpy.py file to spawn a shell.
nano numpy.py
import os
os.system("/bin/bash")
It is important to first set up execute permissions on the spoofed numpy.py file:
chmod +x numpy.py
Now we can carry out the python library hijack and spawn a shell as user ben without knowing their password by running the following command:
sudo -u ben /usr/bin/python3 /home/vulnerableuser/checkmypermissions.py
Project Learnings
Learning #1
I learned that Visudo is a special editor within Linux to change the sudoers file /etc/sudoers.
It helps check formatting to avoid any errors or crashes from poorly written lines. The sudoers file allows the root user to granularize user permissions with the sudoers file on Linux.
Learning #2
Granting run as another user file permissions can expose a machine to library hijacking vulnerabilities.
Running sudo -l can help expose special user file permissions when enumerating for attack vectors to execute privilege escalation.
Learning #3
I found that it is helpful to compile a custom shortlist of Python and bash commands new to me for each project. I borrowed this strategy from my experience with language learning.
Over the years, I’ve improved my Mandarin by taking notes on new vocabulary words and grammar patterns. When working on a new topic area I would always create my own custom grammar and vocabulary lists for reference.
I’ve found that the simple act of focusing on recording a list helps to cement my learning and creates a nice reference for later use.
Quick Fix: Python raises the ModuleNotFoundError: No module named 'dotenv' when it cannot find the library dotenv. The most frequent source of this error is that you haven’t installed dotenv explicitly with pip install python-dotenv. Alternatively, you may have different Python versions on your computer, and dotenv is not installed for the particular version you’re using.
Other variants of the installation statement you should try if this doesn’t work are (in that order):
You’ve just learned about the awesome capabilities of the dotenv library and you want to try it out, so you start your code with the following statement:
from dotenv import load_dotenv load_dotenv() # take environment variables from .env.
This is supposed to import the dotenv library into your (virtual) environment. However, it only throws the following ImportError: No module named dotenv:
>>> import dotenv
Traceback (most recent call last): File "<pyshell#6>", line 1, in <module> import dotenv
ModuleNotFoundError: No module named 'dotenv'
Solution Idea 1: Install Library dotenv
The most likely reason is that Python doesn’t provide dotenv in its standard library. You need to install it first!
To fix this error, you can run the following command in your Windows shell:
$ pip install python-dotenv
This simple command installs dotenv in your virtual environment on Windows, Linux, and MacOS. It assumes that your pip version is updated. If it isn’t, use the following two commands in your terminal, command line, or shell (there’s no harm in doing it anyways):
Note: Don’t copy and paste the $ symbol. This is just to illustrate that you run it in your shell/terminal/command line.
Solution Idea 2: Fix the Path
The error might persist even after you have installed the dotenv library. This likely happens because pip is installed but doesn’t reside in the path you can use. Although pip may be installed on your system the script is unable to locate it. Therefore, it is unable to install the library using pip in the correct path.
To fix the problem with the path in Windows follow the steps given next.
Step 1: Open the folder where you installed Python by opening the command prompt and typing where python
Step 2: Once you have opened the Python folder, browse and open the Scripts folder and copy its location. Also verify that the folder contains the pip file.
Step 3: Now open the Scripts directory in the command prompt using the cd command and the location that you copied previously.
Step 4: Now install the library using pip install python-dotenv command. Here’s an analogous example:
After having followed the above steps, execute our script once again. And you should get the desired output.
Other Solution Ideas
The ModuleNotFoundError may appear due to relative imports. You can learn everything about relative imports and how to create your own module in this article.
You may have mixed up Python and pip versions on your machine. In this case, to install dotenv for Python 3, you may want to try python3 -m pip install python-dotenv or even pip3 install python-dotenv instead of pip install python-dotenv
If you face this issue server-side, you may want to try the command pip install – user python-dotenv
If you’re using Ubuntu, you may want to try this command: sudo apt install python-dotenv
You can also check out this article to learn more about possible problems that may lead to an error when importing a library.
Understanding the “import” Statement
import dotenv
In Python, the import statement serves two main purposes:
Search the module by its name, load it, and initialize it.
Define a name in the local namespace within the scope of the import statement. This local name is then used to reference the accessed module throughout the code.
What’s the Difference Between ImportError and ModuleNotFoundError?
What’s the difference between ImportError and ModuleNotFoundError?
Python defines an error hierarchy, so some error classes inherit from other error classes. In our case, the ModuleNotFoundError is a subclass of the ImportError class.
You can see this in this screenshot from the docs:
You can also check this relationship using the issubclass() built-in function:
Specifically, Python raises the ModuleNotFoundError if the module (e.g., dotenv) cannot be found. If it can be found, there may be a problem loading the module or some specific files within the module. In those cases, Python would raise an ImportError.
If an import statement cannot import a module, it raises an ImportError. This may occur because of a faulty installation or an invalid path. In Python 3.6 or newer, this will usually raise a ModuleNotFoundError.
Related Videos
The following video shows you how to resolve the ImportError:
How to Fix “ModuleNotFoundError: No module named ‘dotenv’” in PyCharm
If you create a new Python project in PyCharm and try to import the dotenv library, it’ll raise the following error message:
Traceback (most recent call last): File "C:/Users/.../main.py", line 1, in <module> import dotenv
ModuleNotFoundError: No module named 'dotenv' Process finished with exit code 1
The reason is that each PyCharm project, per default, creates a virtual environment in which you can install custom Python modules. But the virtual environment is initially empty—even if you’ve already installed dotenv on your computer!
Here’s a screenshot exemplifying this for the pandas library. It’ll look similar for dotenv.
The fix is simple: Use the PyCharm installation tooltips to install Pandas in your virtual environment—two clicks and you’re good to go!
First, right-click on the pandas text in your editor:
Second, click “Show Context Actions” in your context menu. In the new menu that arises, click “Install Pandas” and wait for PyCharm to finish the installation.
The code will run after your installation completes successfully.
As an alternative, you can also open the Terminal tool at the bottom and type:
In this article, we will use PyTorch to build a working neural network. Specifically, this network will be trained to recognize handwritten numerical digits using the famous MNIST dataset.
The code in this article borrows heavily from the PyTorch tutorial “Learn the Basics”. We do this for several reasons.
First, that tutorial is pretty good at demonstrating the essentials for getting a working neural network.
Second, just like importing libraries, it’s good to not reinvent the wheel when you don’t have to.
Third, when building your own network, it is very helpful to start with something that is known to work, then modify it to your needs.
Knowledge Background
This article assumes the reader has some necessary background:
Familiarity with Matplotlib. While this is not necessary to follow along, it is necessary if you want to be able to view image data yourself on your own datasets in the future (and you will want to be able to do this).
You can run PyTorch on your own machine, or you can run it on publically available computer systems.
We will be running this exercise using Google Colab, which allows running world-class computing capability, all accessible for free.
This article will cover all the necessary steps to build and test a working neural network using the PyTorch library.
PyTorch provides a framework that makes building, training, and using neural networks easier. Also under the hood, it is written using the very fast C++ language, so that those neural networks can provide world-class performance while using the popular Python language as the interface to create those networks.
Neural networks and the PyTorch library are rich subjects. So while we will cover all the necessary steps, each step will just scratch the surface of its respective subject.
For example, we will get the image data from datasets built into the PyTorch library. However, the user will eventually want to use neural networks on their own data, so the users will need to learn how to build and work with their own datasets.
So for each of these steps, the user will want to learn more on each subject to become a proficient PyTorch user.
Nevertheless, by the end of this article, you will have built your own working neural network, so you can be sure you will know how to do it!
Further learning will enrich those abilities. Throughout the article, we will point out some of the other things you will eventually want to learn for each step.
Here are the steps we will be taking:
Import necessary libraries.
Acquire the data.
Review the data to understand it.
Create data loaders for loading the data into the network.
Design and create the neural network.
Specify the loss measure and the optimizer algorithm.
Specify the training and testing functions.
Train and test the network using the specified functions.
Step 1: Import Necessary Libraries
Before we do anything, we will want to set up our runtime to use the GPU (again, assuming here you are using Colab).
Click on “Runtime” in the top menu bar, and then choose “Change runtime type” from the dropdown. Then from the window that pops up choose “GPU” under “Hardware accelerator”, and then click “Save”.
Next, we will need to import a number of libraries:
We will import the torch library, making PyTorch available for use.
From the torch module we will import the nn library, which is important for building the neural network.
From the torchvision module we will import the datasets library, which will help provide the image datasets.
From the data utilities module, we will import the DataLoader library. Data loaders help load data into the network.
From the torchvision.transforms module we will import the ToTensor library. This converts the image data into tensors so that they are ready to be processed through the network.
Here is the code importing the needed modules:
import torch
from torch import nn
from torchvision import datasets
from torch.utils.data import DataLoader
from torchvision.transforms import ToTensor
Step 2: Acquire the Data
As mentioned before, in this exercise, we will be getting the MNIST data as available directly through PyTorch libraries. This is the quickest and easiest approach to getting the data.
If you wanted to get the original datasets they are available at:
Even though we will get the data through the PyTorch libraries, it can still be helpful to review this page, as it provides some useful information about the dataset. (However we will provide everything you need to understand this dataset in the article).
Note: Firefox has trouble accessing this page, for some reason requiring a login to access it. Either view it using another browser, or view it as recorded on the Internet Archive Wayback Machine.
There are multiple datasets available through the PyTorch dataset libraries. Here are PyTorch webpages linking to Image Datasets, Text Datasets, and Audio Datasets.
To get data from a PyTorch dataset we create an instance from the respective dataset class. Here is the format:
dataset_instance = DatasetClass(parameters)
This creates a dataset object, and downloads the data. The data is then available by working with the dataset object.
Here is the code to create our MNIST datasets:
# Download MNIST data, put it in pytorch dataset
mnist_data = datasets.MNIST( root='mnist_nn', train=True, download=True, transform=ToTensor()
)
The root parameter specifies the directory where the downloaded data will be placed.
The train parameter determines whether training or testing data is downloaded.
The download=True parameter confirms the data should be downloaded if it hasn’t been already.
The transform parameter converts the data into tensors, in this case.
What parameters are available vary from dataset to dataset, as does how the data is structured, so refer to the dataset web pages mentioned above to review the details of what is available and needed.
While this method of getting data is convenient and easy, remember that you will eventually want to work with your own data, so eventually, you will want to learn how to create your own datasets.
Also, not all datasets contain images with uniform image size, so images may need to be cropped or stretched to fit the fixed number of input neurons.
Also, other transformations can be helpful as well.
For example, you can effectively expand your dataset by including subcrops from your original dataset as additional images to train on. So data transformations is something else you will want to learn that you might use at this stage in the process.
Step 3: Review the Dataset
Now that we have downloaded the data and created a dataset, let’s review the dataset to understand its contents and structure.
The type() function shows that our dataset is an object of the MNIST dataset class.
Conveniently, PyTorch datasets have been designed to be indexed like lists. Let’s take advantage of this and use the len() function to learn something about our datasets:
len(mnist_data)
# 60000
len(mnist_test_data)
# 10000
So our training dataset contains 60000 items, and our test dataset contains 10000 items, consistent with the number of images specified to be in each respective dataset.
Let’s use the type() and len() functions to examine the first item in the training dataset:
type(mnist_data[0])
# tuple
len(mnist_data[0])
# 2
So the items in the datasets are tuples containing 2 items.
Let’s use the type() function to learn about the first item in the tuple:
type(mnist_data[0][0])
# torch.Tensor
So the first item in the tuple is a tensor, likely some image data.
Let’s examine the shape attribute of the tensor to understand its shape:
mnist_data[0][0].shape
# torch.Size([1, 28, 28])
This is consistent with the 28*28 pixel structure of the image data, plus one additional dimension containing the entire image data.
Let’s examine the second item in the tuple:
type(mnist_data[0][1])
# int
mnist_data[0][1]
# 5
So the second item is the integer '5', apparently the label for an image of the digit '5'.
Let’s use Matplotlib to view the image:
import matplotlib.pyplot as plt
plt.imshow(mnist_data[0][0], cmap='gray')
Output:
TypeError Traceback (most recent call last)
<ipython-input-14-3e7278364eac> in <module>
----> 1 plt.imshow(mnist_data[0][0], cmap='gray') /usr/local/lib/python3.7/dist-packages/matplotlib/pyplot.py in imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, data, **kwargs) 2649 filternorm=filternorm, filterrad=filterrad, imlim=imlim, 2650 resample=resample, url=url, **({"data": data} if data is not
-> 2651 None else {}), **kwargs) 2652 sci(__ret) 2653 return __ret /usr/local/lib/python3.7/dist-packages/matplotlib/__init__.py in inner(ax, data, *args, **kwargs) 1563 def inner(ax, *args, data=None, **kwargs): 1564 if data is None:
-> 1565 return func(ax, *map(sanitize_sequence, args), **kwargs) 1566 1567 bound = new_sig.bind(ax, *args, **kwargs) /usr/local/lib/python3.7/dist-packages/matplotlib/cbook/deprecation.py in wrapper(*args, **kwargs) 356 f"%(removal)s. If any parameter follows {name!r}, they " 357 f"should be pass as keyword, not positionally.")
--> 358 return func(*args, **kwargs) 359 360 return wrapper /usr/local/lib/python3.7/dist-packages/matplotlib/cbook/deprecation.py in wrapper(*args, **kwargs) 356 f"%(removal)s. If any parameter follows {name!r}, they " 357 f"should be pass as keyword, not positionally.")
--> 358 return func(*args, **kwargs) 359 360 return wrapper /usr/local/lib/python3.7/dist-packages/matplotlib/axes/_axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs) 5624 resample=resample, **kwargs) 5625 -> 5626 im.set_data(X) 5627 im.set_alpha(alpha) 5628 if im.get_clip_path() is None: /usr/local/lib/python3.7/dist-packages/matplotlib/image.py in set_data(self, A) 697 or self._A.ndim == 3 and self._A.shape[-1] in [3, 4]): 698 raise TypeError("Invalid shape {} for image data"
--> 699 .format(self._A.shape)) 700 701 if self._A.ndim == 3: TypeError: Invalid shape (1, 28, 28) for image data
Oops, that extra one-item dimension (containing the whole image) is causing us problems. We can use the squeeze() method on the tensor to get rid of any one-element dimensions, and instead return a two-dimensional 28*28 tensor, instead of the three-dimensional tensor we had before.
Let’s try again:
plt.imshow(mnist_data[0][0].squeeze(), cmap='gray')
# <matplotlib.image.AxesImage at 0x7f5b5e336150>
Well, it’s a little sloppy, but that’s plausibly a number '5'. (This is reasonable to expect from a hand-written digit!).
So it looks like each item in the dataset is a tuple containing an image (in tensor format) and its corresponding label.
Let’s use Matplotlib to look at the first 10 images, and title each image with its corresponding label:
fig, axs = plt.subplots(2, 5, figsize=(8, 5))
for a_row in range(2): for a_col in range(5): img_no = a_row*5 + a_col img = mnist_data[img_no][0].squeeze() img_tgt = mnist_data[img_no][1] axs[a_row][a_col].imshow(img, cmap='gray') axs[a_row][a_col].set_xticks([]) axs[a_row][a_col].set_yticks([]) axs[a_row][a_col].set_title(img_tgt, fontsize=20)
plt.show()
So now we have a clear understanding of how our dataset is structured and what the data looks like. Much of this is explained in the dataset description page, but this kind of analysis is often very useful for getting a precise understanding of the dataset that might not be clear from the description.
Step 4: Create Dataloaders
Datasets make the data available for processing.
However, typically, we will want to process using randomized mini-batches from the dataset.
Data loaders make this easy. Dataloaders are iterables, and you’ll see later that every time you iterate a dataloader it returns a randomized minibatch from the dataset that can be processed through the neural network.
Let’s create some dataloader objects from our datasets:
So we have created two data loaders, one for the training dataset, and one for the test dataset.
The batch_size parameter specifies the number of image/label pairs in the minibatch that the dataloader will return for each iteration. The shuffle parameter determines whether or not the mini-batches are randomized.
Step 5: Design and Create the Neural Network
Check for GPU
We are about to design and create the neural network, but first, let’s check if a GPU is available.
One of the advantages PyTorch has as a neural network framework is that it supports the use of a GPU. The use of a GPU will implement parallel processing to greatly speed up computation.
Depending on the problem, at least an order of magnitude faster processing can be achieved.
Use of a GPU with PyTorch is very easy. First, use the function torch.cuda.is_available() to test if a GPU is available and properly configured for use by PyTorch (PyTorch uses the CUDA framework for using the GPU).
If a GPU is available, we will send the model and the data tensors to the GPU for processing.
The following tests for availability of a GPU, then sets a variable device to either 'cpu' or 'cuda' depending on what is available.
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using {device} device")
# Using cuda device
Create the Neural Network
Now let’s design and create the neural network. We do this by creating a class, which we have chosen to call NeuralNet, which is a subclass of the nn.Module library.
Here is the code to specify and then create our neural network:
class NeuralNet(nn.Module): def __init__(self): super().__init__() # Required to properly initialize class, ensures inheritance of the parent __init__() method self.flat_f = nn.Flatten() # Creates function to smartly flatten tensor self.neur_net = nn.Sequential( nn.Linear(28*28, 512), nn.ReLU(), nn.Linear(512, 256), nn.ReLU(), nn.Linear(256,10) ) def forward(self, x): x = self.flat_f(x) logits = self.neur_net(x) return logits model = NeuralNet().to(device)
There are a number of important details to review in this code.
First, our neural network definition class must have two methods included: an __init__() method, and a forward() method.
Classes in Python routinely include an __init__() method to initialize variables and other things in the object that is created. The class must also include a forward() method, which tells PyTorch how to process the data during the forward pass of the data.
Let’s go over each of these in more detail.
Creating the Model: __init__() Method
First, within the __init__() method note the super().__init__() command. When we create a subclass it inherits the parent class variables and methods.
However, when we write an __init__() method in the subclass, that overrides inheritance of the __init__() method from the parent class.
However there are features in the parent class’ __init__() that our class needs to inherit. The super()__.init__() command achieves this. In effect, it says “include the parent class __init__() within our child class”.
To make a long story short, this is necessary to properly initialize our child class, by including some things needed from the parent nn.Module class.
Next, note creating a function from the nn.Flatten() function. Even though our data is a 28×28 pixel two-dimensional image, the processing still works if we convert it into a one-dimensional vector, stacking row by row next to one another to form a 28×28 = 784 element vector (in fact making this change is a common choice).
The flatten() function achieves this. However, the standard flatten() (note the lower case 'f') function will flatten everything, turning a 100 image minibatch tensor of shape (100, 1, 28, 28) into a single vector of shape (78400).
Instead, if we create a function from the nn.Flatten() function (note the upper case 'F'), this is smart enough to know to eliminate the single-element dimension and merge the last two dimensions, resulting in a tensor of shape (100, 784), representing a list of 100 vectors of 784 elements.
Note: double-check to make sure your function is flattening properly. If not, the Flatten() function can include some parameters that specify which dimensions to flatten. See documentation for details.
The last thing we do in the __init__() method is specify the neural network structure using the nn.Sequential() function.
Here we list the neural network layers in sequence from beginning to end.
First, we list an input layer of 28×28=784 neurons, connecting through linear (weights * input + bias) connections to 512 neurons. These 512 neurons then pass data through a non-linear ReLU activation function layer.
Those signals then go through another linear layer connecting 512 neurons to 256 neurons. These signals then go through another ReLU activation function layer. Finally, the signals go through a final linear layer connecting the 256 neurons to 10 final output neurons.
'ReLU' stands for 'Rectified Linear Unit'. It is one of many non-linear activation functions which can be chosen.
It is defined as:
f(x) = x, if x>=0
else f(x) = 0
Here is a graph of the ReLU function:
Creating the Model: forward() Method
The second required method for our class is the forward() method.
As mentioned the forward() method tells PyTorch how to process the data during the forward pass. Here we first flatten our tensor using the flatten function we defined previously under __init__().
Then we pass the tensor through the self.neur_net() function we defined previously using the nn.Sequential() function. Finally, the results are returned.
Important point: the programmer will NOT be using forward() method in any classes or functions, it is just for PyTorch’s use. PyTorch expects such a method, so it must be written, but the programmer will not directly use it in any subsequent code.
Finally, we create the neural network (here named 'model') by creating an instance of our NeuralNet() class. In addition, we move the model to the GPU (if available) by including the .to(device) method.
Finally, we can choose to print the model to examine the neural network object we have built:
Next, we’ll need to specify our loss function and our optimizer algorithm.
Choosing Cross Entropy Loss
Recall the loss function measures how far the model’s guess is from the correct answer for a given input. Adjusting weights and biases to minimize loss is how neural networks learn (see the Finxter article “How Neural Networks Learn” for details.).
There are multiple choices of loss functions available, and learning about these various functions is something you will want to do, because which loss choice is most suitable depends on the particular kind of problem you are solving.
In this case, we are sorting images into multiple categories.
One of the most suitable loss choices for this case is cross-entropy loss. Cross entropy is an idea taken from information theory, and it is a measure of how many extra bits must be sent when sending a message using a sub-optimized code.
This is beyond the scope of this exercise, but we can understand its usefulness to our situation if we examine the calculation involved:
That is, for each category multiply the true probability t by the log of the model’s estimated probability p, and add them all up.
Of course, t is zero for each incorrect category, and 1 for the correct category.
Consequently, for any given image, just the correct category is selected to contribute to the loss calculation, and that loss is the negative of the log of the probability estimate.
Recall this is what the log() function looks like:
Since the network provides a probability estimate we are only interested in the interval (0,1]. Here is what the negative of the log() looks like over that interval:
So the loss is very large when the network gives a low probability estimate (near zero) for the correct category, and the loss is lowest (near zero) when the network gives a high probability estimate (near 1.0) for the correct category.
Here is the code specifying cross entropy loss as the loss function:
loss_fn = nn.CrossEntropyLoss()
Choosing Optimizer Algorithm
We also need to choose the optimizer algorithm. This is the method used to minimize the loss through training. Multiple different optimizers may be chosen, and you will want to learn about the various optimizers available.
All are variations on gradient descent.
For example, some include extinction of the learning rate; others include momentum that helps drive loss away from local minima.
In our case, we will choose plain-old vanilla stochastic gradient descent. Here is the code specifying the optimizer and its learning rate:
Now we define functions for training and testing the neural network.
Training Function
Here is the code specifying the training function:
def train_nn(dataloader, model, loss_fn, optimizer): size = len(dataloader.dataset) for batch, (X, y) in enumerate(dataloader): X, y = X.to(device), y.to(device) # For each image in batch X, compute prediction pred = model(X) # Compute average loss for the set of images in batch loss = loss_fn(pred, y) # Backpropagation optimizer.zero_grad() # Zero gradients loss.backward() # Computes gradients optimizer.step() # Update weights, biases according to gradients, factored by learning rate if batch % 100 == 0: # Report progress every 100 batches loss, current = loss.item(), batch * len(X) print(f"loss: {loss:>7f} [{current:>5d}/{size:>5d}]")
We pass into the function the dataloader, model, loss function, and optimizer objects.
The function then loops over minibatches from the dataloader.
For each loop, a minibatch of the input images X and the labels y is retrieved and then moved to the GPU (if available).
Then the neural network model calculates predictions from the input images X. These predictions and the correct labels y are used to calculate the loss (note this loss is a single number that is the average loss for the minibatch).
Once the loss is calculated, the function can adjust weights and biases (backpropagate) in three code steps.
First, gradient attributes are zeroed out using optimizer.zero_grad() (PyTorch defaults to accumulating gradient calculations, so they need to be zeroed out on each iteration of the loop, or else they’ll keep accumulating data).
Then the gradients are calculated using loss.backward(). Finally, weights and biases are updated according to the gradients using optimizer.step().
Finally, a small section is included to report progress every 100 batches. This prints out the current loss, and how many images of the total images have been completed.
Testing Function
Here is the code specifying the testing function:
def test_loop(dataloader, model, loss_fn): # After each epoch, test training results (report categorizing accuracy, loss) size = len(dataloader.dataset) # Number of image/label pairs in dataset num_batches = len(dataloader) test_loss, correct = 0, 0 # Initialize variables tracking loss and accuracy during test loop with torch.no_grad(): # Disable gradient tracking - reduces resource use and speeds up processing for X, y in dataloader: X, y = X.to(device), y.to(device) pred = model(X) # Get predictions from the neural network based on input minibatch X test_loss += loss_fn(pred, y).item() # Accumulate loss values during loop through dataset correct += (pred.argmax(1) == y).type(torch.float).sum().item() # Accumulate correct predictions during loop through dataset test_loss /= num_batches # Calculate average loss correct /= size # Calculate accuracy rate print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n") # Report test results
This function tests the accuracy of the network using the test data.
First, we pass in the testing data loader, the model, and the loss function (for testing loss). Then the function initializes several variables, especially test_loss and correct for accumulating test results during the test loop.
The function does the next few steps within a with torch.no_grad(): subsection.
Here is why: PyTorch stores calculations from the forward pass for later use during the backpropagation gradient calculations.
The torch.no_grad() method turns that off while in this with subsection, since there will be only a forward pass during the testing. This saves resources and speeds up processing. You will want to do the same thing once you have a trained network that is used for classifying in production.
After leaving the with subsection the calculation-storing feature automatically resumes.
Note: be aware that storing calculations is turned on (requires_grad=True) because we are using Modules from the nn library (Linear, ReLU). Otherwise, PyTorch tensors default to requires_grad=False.
Then the function uses a for loop to iterate through the minibatches of the test dataloader. For each iteration, the neural network model computes predictions from the minibatch of images. The loss is calculated for the minibatch, which is then accumulated in test_loss.
Then the number of correct predictions for the minibatch is found as follows: first note that pred is a set of 10-element vectors, with each element an estimate of the probability of that element index being the correct prediction.
The .argmax(1) method returns the index of the largest estimate (the number 1 in the argmax() argument indicates which dimension to use for the operation). This list (tensor) of indices is compared to the list (tensor) of correct labels in y.
This results in a list (tensor) containing True where there is a match, and False otherwise. The type(torch.float) method converts these into floating point 1’s and 0’s.
The sum() method adds all the elements together. Then finally, the .item() method converts the totaled one-element tensor into a raw number (scalar).
Finally, we have the total number of correct predictions for that batch, which is added to the correct variable that accumulates the total number of correct predictions as the for loop iterates through the dataloader.
Train and Test the Network
Now we have written enough code, we can write a small main program loop to train and test the network. We specify how many epochs we wish to run, then we loop through those epochs, training and testing the network for each one.
Here is the code:
# The main program! epochs = 5
for t in range(epochs): print(f"Epoch {t+1}\n-------------------------------") train_nn(mnist_train_dl, model, loss_fn, optimizer) test_loop(mnist_test_dl, model, loss_fn)
print("Done!")
After just 5 epochs, the accuracy isn’t very good yet, but we can see that things are moving in the right direction.
Obviously, if we wanted to get good performance we would need to train for more epochs. Figuring out how much to train (being careful not to overfit!) is something a neural network engineer has to work out.
Reviewing the Big Picture
It may seem like we have gone over a lot, and we have, but if you step back and look at the big picture there isn’t a lot here.
It may seem like a lot because we have reviewed everything in detail to make sure we convey full understanding.
However, to gain some perspective, let’s show all the essential code, without all the extra description and explanation (note, we’re also skipping the code here used to review the dataset):
Import Necessary Libraries
import torch
from torch import nn
from torchvision import datasets
from torch.utils.data import DataLoader
from torchvision.transforms import ToTensor
Acquire the Data
# Download MNIST data, put it in pytorch dataset
mnist_data = datasets.MNIST( root='mnist_nn', train=True, download=True, transform=ToTensor()
)
def train_nn(dataloader, model, loss_fn, optimizer): size = len(dataloader.dataset) for batch, (X, y) in enumerate(dataloader): X, y = X.to(device), y.to(device) # For each image in batch X, compute prediction pred = model(X) # Compute average loss for the set of images in batch loss = loss_fn(pred, y) # Backpropagation optimizer.zero_grad() # Zero gradients loss.backward() # Computes gradients optimizer.step() # Update weights, biases according to gradients, factored by learning rate if batch % 100 == 0: # Report progress every 100 batches loss, current = loss.item(), batch * len(X) print(f"loss: {loss:>7f} [{current:>5d}/{size:>5d}]")
def test_loop(dataloader, model, loss_fn): # After each epoch, test training results (report categorizing accuracy, loss) size = len(dataloader.dataset) # Number of image/label pairs in dataset num_batches = len(dataloader) test_loss, correct = 0, 0 # Initialize variables tracking loss and accuracy during test loop with torch.no_grad(): # Disable gradient tracking - reduces resource use and speeds up processing for X, y in dataloader: X, y = X.to(device), y.to(device) pred = model(X) # Get predictions from the neural network based on input minibatch X test_loss += loss_fn(pred, y).item() # Accumulate loss values during loop through dataset correct += (pred.argmax(1) == y).type(torch.float).sum().item() # Accumulate correct predictions during loop through dataset test_loss /= num_batches # Calculate average loss correct /= size # Calculate accuracy rate print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n") # Report test results
Train and Test the Network
# The main program! epochs = 5
for t in range(epochs): print(f"Epoch {t+1}\n-------------------------------") train_nn(mnist_train_dl, model, loss_fn, optimizer) test_loop(mnist_test_dl, model, loss_fn)
print("Done!")
Really we have written just a few dozen lines of code, comparable to the size program a hobbyist programmer might write.
Yet we’ve built a world-class neural network that converts hand-written digits to numbers a computer can work with. That’s pretty amazing!
Of course, this is all possible thanks to the efforts of the many engineers who wrote the many more lines of code within PyTorch. Thank you to all of you who have contributed to PyTorch!
This is another example of achieving great things by standing on the shoulders of giants!
Saving and Reloading the Network
We have built, trained, and tested a neural network, and that’s great. But really, the point of training a neural network is to put it to use. To support that, we need to be able to save and reload the network for later use.
Use the following code to save the weights and biases of your neural network (note: the common convention is to save these files with extension .pt or .pth):
To reload, first create an instance of your neural network (make sure you have access to the class/neural network you originally specified). In our example:
user_model = NeuralNet().to(device)
Then load the new instance with your saved weights and biases:
Some of the modules perform differently when in training rather than when in use.
Specifically, when in training mode, some of them implement various regularization methods which are used to resist the onset of overfitting.
These methods may include some randomness and can cause the network to give inconsistent results. To avoid this, make sure you are in evaluation mode and not training mode:
As you can see this command conveniently reports the neural network structure.
Let’s make sure our reloaded network works.
It would be best to test with some new handwritten digits, but for the sake of convenience lets just test it with the first ten test images (especially since the network was not trained very heavily).
Let’s look at these first ten images in the test dataset:
fig, axs = plt.subplots(2, 5, figsize=(8, 5))
for a_row in range(2): for a_col in range(5): img_no = a_row*5 + a_col img = mnist_test_data[img_no][0].squeeze() img_tgt = mnist_test_data[img_no][1] axs[a_row][a_col].imshow(img, cmap='gray') axs[a_row][a_col].set_xticks([]) axs[a_row][a_col].set_yticks([]) axs[a_row][a_col].set_title(img_tgt, fontsize=20)
plt.show()
Now let’s see if the network detects these images properly:
def eval_image(model, imgno): testimg = mnist_test_data[imgno][0] # assign first image to variable 'testimg' testimg = testimg.to(device) # move image data to GPU logits = model(testimg) # run image through network return logits.argmax().item() # argmax id's value, returns it for img_no in range(10): img_val = eval_image(model, img_no) print(img_val)
Output:
7
2
1
0
4
1
7
9
6
7
The results are not perfect, but for an incompletely trained network that’s not bad! The few failure are plausible given the incomplete training. Our network works with the saved and reloaded weights and biases!
Conclusion
We hope you have found this article educational, and we hope it inspires you to go and build your own working neural networks using PyTorch!
Quick Fix: Python raises the ModuleNotFoundError: No module named 'ffmpeg' when it cannot find the library ffmpeg. The most frequent source of this error is that you haven’t installed ffmpeg explicitly with pip install ffmpeg-python or even pip3 install ffmpeg-python for Python 3. Alternatively, you may have different Python versions on your computer, and ffmpeg is not installed for the particular version you’re using.
Problem Formulation
You’ve just learned about the awesome capabilities of the ffmpeg library and you want to try it out, so you start your code with the following statement:
The first line is supposed to import the ffmpeg library into your (virtual) environment. However, it only throws the following ImportError: No module named ffmpeg:
>>> import ffmpeg
Traceback (most recent call last): File "<pyshell#6>", line 1, in <module> import ffmpeg
ModuleNotFoundError: No module named 'ffmpeg'
Solution Idea 1: Install Library ffmpeg
The most likely reason is that Python doesn’t provide ffmpeg in its standard library. You need to install it first!
To fix this error, you can run the following command in your Windows shell:
$ pip install ffmpeg-python
This simple command installs ffmpeg in your virtual environment on Windows, Linux, and MacOS. It assumes that your pip version is updated. If it isn’t, use the following two commands in your terminal, command line, or shell (there’s no harm in doing it anyways):
Note: Don’t copy and paste the $ symbol. This is just to illustrate that you run it in your shell/terminal/command line.
Solution Idea 2: Fix the Path
The error might persist even after you have installed the ffmpeg library. This likely happens because pip is installed but doesn’t reside in the path you can use. Although pip may be installed on your system the script is unable to locate it. Therefore, it is unable to install the library using pip in the correct path.
To fix the problem with the path in Windows follow the steps given next.
Step 1: Open the folder where you installed Python by opening the command prompt and typing where python
Step 2: Once you have opened the Python folder, browse and open the Scripts folder and copy its location. Also verify that the folder contains the pip file.
Step 3: Now open the Scripts directory in the command prompt using the cd command and the location that you copied previously.
Step 4: Now install the library using pip install ffmpeg-python command. Here’s an analogous example:
After having followed the above steps, execute our script once again. And you should get the desired output.
Other Solution Ideas
The ModuleNotFoundError may appear due to relative imports. You can learn everything about relative imports and how to create your own module in this article.
You may have mixed up Python and pip versions on your machine. In this case, to install ffmpeg for Python 3, you may want to try python3 -m pip install ffmpeg-python or even pip3 install ffmpeg-python instead of pip install ffmpeg-python
If you face this issue server-side, you may want to try the command pip install – user ffmpeg-python
If you’re using Ubuntu, you may want to try this command: sudo apt install ffmpeg-python
You can also check out this article to learn more about possible problems that may lead to an error when importing a library.
Understanding the “import” Statement
import ffmpeg
In Python, the import statement serves two main purposes:
Search the module by its name, load it, and initialize it.
Define a name in the local namespace within the scope of the import statement. This local name is then used to reference the accessed module throughout the code.
What’s the Difference Between ImportError and ModuleNotFoundError?
What’s the difference between ImportError and ModuleNotFoundError?
Python defines an error hierarchy, so some error classes inherit from other error classes. In our case, the ModuleNotFoundError is a subclass of the ImportError class.
You can see this in this screenshot from the docs:
You can also check this relationship using the issubclass() built-in function:
Specifically, Python raises the ModuleNotFoundError if the module (e.g., ffmpeg) cannot be found. If it can be found, there may be a problem loading the module or some specific files within the module. In those cases, Python would raise an ImportError.
If an import statement cannot import a module, it raises an ImportError. This may occur because of a faulty installation or an invalid path. In Python 3.6 or newer, this will usually raise a ModuleNotFoundError.
Related Videos
The following video shows you how to resolve the ImportError:
How to Fix “ModuleNotFoundError: No module named ‘ffmpeg’” in PyCharm
If you create a new Python project in PyCharm and try to import the ffmpeg library, it’ll raise the following error message:
Traceback (most recent call last): File "C:/Users/.../main.py", line 1, in <module> import ffmpeg
ModuleNotFoundError: No module named 'ffmpeg' Process finished with exit code 1
The reason is that each PyCharm project, per default, creates a virtual environment in which you can install custom Python modules. But the virtual environment is initially empty—even if you’ve already installed ffmpeg on your computer!
Here’s a screenshot exemplifying this for the pandas library. It’ll look similar for ffmpeg-python.
The fix is simple: Use the PyCharm installation tooltips to install Pandas in your virtual environment—two clicks and you’re good to go!
First, right-click on the pandas text in your editor:
Second, click “Show Context Actions” in your context menu. In the new menu that arises, click “Install Pandas” and wait for PyCharm to finish the installation.
The code will run after your installation completes successfully.
As an alternative, you can also open the Terminal tool at the bottom and type:
Problem: How to split a string using an empty string as a separator?
Related Read: Python list() — A Simple Guide with Video
Question: How to fix the 
Summary: Split the string using split and then use len to count the results.
Recommended Tutorial: 
Recommended Tutorial: 
Note: Don’t copy and paste the 
