As your projects grow in size and complexity, you may need more computing power than your local machine can provide. You may require secure, centralised storage solutions that scale seamlessly. Cloud platforms offer scalable resources and specialised tools for data science.
Cloud platforms offer several advantages for data science:
The ability to scale compute resources is particularly valuable for data scientists working with large datasets or computationally intensive models. Rather than investing in expensive hardware that might sit idle most of the time, cloud platforms allow you to rent powerful machines when you need them and shut them down when you don’t.
Google Colab provides free access to Python notebooks with GPU and TPU acceleration. It’s an excellent way to get started with cloud-based data science without any financial commitment.
Google Colab is essentially Jupyter notebooks running on Google’s servers, with a few additional features. You can run Python code, create visualisations, and even access GPU and TPU accelerators for free (with usage limits).
The key advantages of Colab include:
Cloud storage services provide an easy way to store and share data:
These services can be used to store datasets, notebooks, and results. They also facilitate collaboration, as you can easily share files with colleagues.
For larger datasets or specialised needs, you’ll want to look at dedicated cloud storage solutions like Amazon S3, Google Cloud Storage, or Azure Blob Storage. These services are designed for scalability and can handle terabytes or even petabytes of data.
For more advanced needs, consider these major cloud platforms:
AWS offers a comprehensive suite of data science tools:
AWS offers a free tier that includes limited access to many of these services, allowing you to experiment before committing financially.
GCP provides similar capabilities:
Azure is particularly well-integrated with Microsoft’s other tools:
Each platform has its strengths, and many organisations use multiple clouds for different purposes. AWS has the broadest range of services, GCP excels in machine learning tools, and Azure integrates well with Microsoft’s enterprise ecosystem.
When selecting cloud services for data science, consider these factors:
A strategic approach is to start with a small project on your chosen platform. This allows you to gain familiarity with the environment before committing to larger workloads.
Let’s create a basic starter project on AWS as an example:
ml.t3.medium (covered by the SageMaker free tier for new accounts)This gives you a fully configured Jupyter environment with access to more computational resources than your local machine likely has. SageMaker comes pre-installed with popular data science libraries and integrates with other AWS services like S3 for storage.
Studio spaces and notebook instances bill by the minute they’re running, not by how much you use them. A notebook you forgot to stop over the weekend is the single most common surprise bill for beginners. Stop the space from the Studio UI when you finish a session.
One of the most important aspects of using cloud platforms is managing costs effectively. A few traps catch almost every beginner:
Practical habits that help:
For example, in AWS you can create a budget with alerts:
When working with cloud platforms, it’s important to remember to shut down resources when you’re not using them to avoid unnecessary charges. Most platforms provide cost management tools to help you monitor and control your spending.
Data security is critical when working in cloud environments:
In practice, the cleanest approach for most data science work is to enable default encryption on the S3 bucket itself (AWS console → Bucket → Properties → Default encryption → SSE-S3 or SSE-KMS). Once that’s set, every object written to the bucket is encrypted at rest automatically, and your notebook code doesn’t need to pass keys at all:
import boto3
from botocore.exceptions import ClientError
def get_data(bucket, key):
# Credentials come from the instance IAM role, not from code.
# Encryption is handled by the bucket's default policy.
s3 = boto3.client('s3')
try:
response = s3.get_object(Bucket=bucket, Key=key)
return response['Body'].read()
except ClientError as e:
print(f"Error accessing data: {e}")
return NoneA few principles to notice:
aws configure to store credentials in ~/.aws/credentials (which is already outside your git repository).Remember that security is a shared responsibility between you and the cloud provider. The provider secures the infrastructure, but you’re responsible for securing your data and applications.
Let’s walk through a complete example of using Google Colab for a data science task. This exercise demonstrates the practical workflow of cloud-based analysis.
In the first cell, load a dataset directly from a URL:
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# Load a sample dataset directly from the web
url = "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"
df = pd.read_csv(url)
# Display basic information
print(f"Dataset shape: {df.shape}")
print(f"\nColumn types:\n{df.dtypes}")
print(f"\nFirst few rows:")
df.head()In subsequent cells, perform your analysis:
# Summary statistics
df.describe()# Create a visualisation
plt.figure(figsize=(10, 6))
sns.scatterplot(data=df, x='bill_length_mm', y='bill_depth_mm',
hue='species', style='island', s=100)
plt.title('Penguin Bill Dimensions by Species and Island')
plt.xlabel('Bill Length (mm)')
plt.ylabel('Bill Depth (mm)')
plt.legend(bbox_to_anchor=(1.05, 1), loc='upper left')
plt.tight_layout()
plt.show()For machine learning tasks, you can enable GPU acceleration:
Then verify GPU availability:
import torch
if torch.cuda.is_available():
print(f"GPU available: {torch.cuda.get_device_name(0)}")
print(f"GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
else:
print("No GPU available - using CPU")Colab notebooks are automatically saved to your Google Drive. You can also:
This exercise demonstrates the core workflow of cloud-based data science: loading data, performing analysis, creating visualisations, and optionally leveraging specialised hardware—all without installing anything on your local machine.
When working with larger datasets, you’ll want to connect cloud storage to your notebooks. Here’s how to mount Google Drive in Colab:
from google.colab import drive
# Mount Google Drive
drive.mount('/content/drive')
# Now you can access files in your Drive
import pandas as pd
df = pd.read_csv('/content/drive/MyDrive/data/my_dataset.csv')For AWS S3 from a Colab notebook, use the boto3 library with credentials supplied through Colab’s secret manager rather than typing them into cells:
# In Colab: click the 🔑 key icon in the left sidebar and add two secrets
# named AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Then:
import os
from google.colab import userdata
import boto3
os.environ["AWS_ACCESS_KEY_ID"] = userdata.get("AWS_ACCESS_KEY_ID")
os.environ["AWS_SECRET_ACCESS_KEY"] = userdata.get("AWS_SECRET_ACCESS_KEY")
s3 = boto3.client("s3", region_name="us-east-1")
s3.download_file("your-bucket", "data.csv", "data.csv")Never paste access keys directly into notebook cells or commit them to a repository. Anything written in a cell gets saved to the .ipynb file and is trivially recoverable. Use Colab secrets, environment variables, or (better) IAM roles for service-to-service access.
If a key does leak, rotate it in the AWS console immediately. Don’t try to scrub the notebook history.
These patterns allow you to work with data stored in various cloud locations while leveraging the computational resources of your cloud notebook environment.
Cloud platforms provide powerful resources for data science, allowing you to scale beyond the limitations of your local machine. Whether you’re using free services like Google Colab or comprehensive platforms like AWS, GCP, or Azure, the cloud offers flexibility, scalability, and specialised tools that can significantly enhance your data science capabilities.
As you grow more comfortable with cloud services, you can explore more advanced features like automated machine learning pipelines, distributed computing, and real-time data processing. The cloud is continuously evolving, with new services and features being added regularly to support data science workflows.
In the upcoming chapters, we’ll explore how to deploy your data science projects to make them accessible to others (Deployment chapter) and how to use containerisation with Docker to ensure your environments are reproducible across local and cloud platforms (Containerisation chapter).