Importing .py files in Google Colab
Jupyter NotebookIpythonGoogle ColaboratoryJupyter Notebook Problem Overview
Is there any way to upload my code in .py files and import them in colab code cells?
The other way I found is to create a local Jupyter notebook then upload it to Colab, is it the only way?
Jupyter Notebook Solutions
Solution 1 - Jupyter Notebook
You can save it first, then import it.
from google.colab import files
src = list(files.upload().values())[0]
open('mylib.py','wb').write(src)
import mylib
Update (nov 2018): Now you can upload easily by
- click at [>] to open the left pane
- choose file tab
- click [upload] and choose your [mylib.py]
- import mylib
Update (oct 2019): If you don't want to upload every time, you can store it in S3 and mount it to Colab, as shown in this gist
Update (apr 2020): Now that you can mount your Google Drive automatically. It is easier to just copy it from Drive than upload it.
- Store
mylib.py
in your Drive - Open a new Colab
- Open the (left)side pane, select
Files
view - Click
Mount Drive
thenConnect to Google Drive
- Copy it by
!cp drive/MyDrive/mylib.py .
import mylib
Solution 2 - Jupyter Notebook
In case anyone else is interested to know how to import files/packages from gdrive inside a google colab. The following procedure worked for me:
1) Mount your google drive in google colab:
from google.colab import drive
drive.mount('/content/gdrive/')
2) Append the directory to your python path using sys:
import sys
sys.path.append('/content/gdrive/mypythondirectory')
Now you should be able to import stuff from that directory!
Solution 3 - Jupyter Notebook
-
You can upload local files to google colab by using upload() function in google.colab.files
-
If you have files on github, then clone the repo using !git clone https://github.com/username/repo_name.git. Then just like in jupyter notebook load it using the magic function %load
%load filename.py
.
Solution 4 - Jupyter Notebook
Based on the answer by Korakot Chaovavanich, I created the function below to download all files needed within a Colab instance.
from google.colab import files
def getLocalFiles():
_files = files.upload()
if len(_files) >0:
for k,v in _files.items():
open(k,'wb').write(v)
getLocalFiles()
You can then use the usual 'import' statement to import your local files in Colab. I hope this helps
Solution 5 - Jupyter Notebook
We can do so.
import sys
import os
py_file_location = "/content/drive/My Drive"
sys.path.append(os.path.abspath(py_file_location))
Now you can import it as module in notebook for that location.
import whatever
Solution 6 - Jupyter Notebook
I face the same problem. After reading numerous posts, I would like to introduce the following solution I finally chose over many other methods (e.g. use urllib
, httpimport
, clone from GitHub, package the modules for installation, etc). The solution utilizes Google Drive API (official doc) for proper authorization.
Advantages:
- Easy and safe (no need for code to handle file operation exceptions and/or additional authorization)
- Module files safeguarded by Google account credentials (no one else can view/take/edit them)
- You control what to upload/access (you can change/revoke access anytime on a file-by-file basis)
- Everything in one place (no need to rely upon or manage another file hosting service)
- Freedom to rename/relocate module files (not path-based and won't break your/other's notebook code)
Steps:
- Save your .py module file to Google Drive - you should have that since you're already using Colab
- Right click on it, "Get shareable link", copy the part after "
id=
" - the file id assigned by Google Drive - Add and run the following code snippets to your Colab notebook:
!pip install pydrive # Package to use Google Drive API - not installed in Colab VM by default
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth # Other necessary packages
from oauth2client.client import GoogleCredentials
auth.authenticate_user() # Follow prompt in the authorization process
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
your_module = drive.CreateFile({"id": "your_module_file_id"}) # "your_module_file_id" is the part after "id=" in the shareable link
your_module.GetContentFile("your_module_file_name.py") # Save the .py module file to Colab VM
import your_module_file_name # Ready to import. Don't include".py" part, of course :)
Side note
Last but not least, I should credit the original contributor of this approach. That post might have some typo in the code as it triggered an error when I tried it. After more reading and troubleshooting my code snippets above worked (as of today on Colab VM OS: Linux 4.14.79).
Solution 7 - Jupyter Notebook
Try this way:
I have a package named plant_seedlings. The package is stored in google drive. What I should do is to copy this package in /usr/local/lib/python3.6/dist-packages/.
!cp /content/drive/ai/plant_seedlings.tar.gz /usr/local/lib/python3.6/dist-packages/
!cd /usr/local/lib/python3.6/dist-packages/ && tar -xzf plant_seedlings.tar.gz
!cd /content
!python -m plant_seedlings
Solution 8 - Jupyter Notebook
Here's my process:
import sys
sys.path.insert(0, '/content/drive/MyDrive/my_folder')
%cd /content/drive/MyDrive/my_folder
%pwd
Now, you can import the module from that path using import my_module
easily
Solution 9 - Jupyter Notebook
You can upload those .py files to Google drive and allow Colab to use to them:
!mkdir -p drive
!google-drive-ocamlfuse drive
All your files and folders in root folder will be in drive
.
Solution 10 - Jupyter Notebook
Below are the steps that worked for me
-
Mount your google drive in google colab
from google.colab import drive drive.mount('/content/drive')
-
Insert the directory
import sys sys.path.insert(0,’/content/drive/My Drive/ColabNotebooks’)
-
check the current directory path
%cd drive/MyDrive/ColabNotebooks %pwd
-
Import your module or file
import my_module
-
If you get the following error 'Name Null is not defined' then do the following
5.1 Download my_module.ipynb from colab as my_module.py file (file->Download .py)
5.2 Upload the *.py file to drive/MyDrive/ColabNotebooks in Google drive
5.3 import my_module will work now
Solution 11 - Jupyter Notebook
It's Jun 2019.
Make sure in the Python package's __init__.py
all related files are imported in order. Push the code to Git or use this code.
for e.g,
from .Boxes import *
from .Circles import *
from .Rectangles import *
...
Don't use Package name in __init__.py
file for importing the files.
in Google colab,
! rm -rf SorghumHeadDetection
! git clone https://github.com/user/amazing-repo-name/
Solution 12 - Jupyter Notebook
you can do this by mount your drive to colab and write some code to put the id of your python file you can find code here importing python file from drive to colab
# Code to read file into colaboratory:
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
#Autheticate E-Mail ID
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
#2.1 Get the file
your_module = drive.CreateFile({"id": 'write your file id here'}) # "your_module_file_id" is the part after "id=" in the shareable link
your_module.GetContentFile("write the file name here") # Save the .py module file to Colab VM
import file_name
from file_name import anything #as classes or functions from your file
Solution 13 - Jupyter Notebook
os.listdir
can be used to view all files in the directory
from google.colab import drive
drive.mount("/content/drive")
import os
path="/content/drive/My Drive/Colab Notebooks"
os.chdir(path)
os.listdir(path)`
Solution 14 - Jupyter Notebook
In my case, the file I was trying to use was called client.py
. This raised a conflict because there's already a library called client
in /usr/local/lib/python3.7/dist-packages/
.
I solved this by uploading the client.py
file to the same Google Drive folder in which the Colab Notebook is saved in and change its name to something unique that doesn't appear in the dist-packages
folder.
In my case, I changed the file name to dfsclient.py
and then just imported it with
import dfsclient
Then I implemented Kamal's answer:
import sys
sys.path.insert(0, '/content/drive/MyDrive/my_folder')
Solution 15 - Jupyter Notebook
This is how I regularly do it:
-
Save my module in the directory. Say MyFile.py in MyModules
-
Define the location of my module:
path_m = '/content/drive/MyDrive/Colab Notebooks/MyModules/'
-
Then I add the path to sys.path:
import sys sys.path.insert(0,path_m)
-
import the module into my Jupyter/Google Colab notebook.
import MyFile
Solution 16 - Jupyter Notebook
A easy way is
- type in from google.colab import files uploaded = files.upload()
- copy the code
- paste in colab cell