Background¶
The TIFF file format (Tagged Image File Format) is an old format dating back to 1992. TIFF are great for high-resolution verbatim raster images. TIFF are still used a bit in high-end photography, but where it has really grown a second life is in digital cartography. The variation called GeoTIFF has been widely adopted as a way to share satellite images and other satellite data.
While the GeoTIFF file format has long been thought of as only suitable for raw data: if you wanted to display it on a map, you’d convert it into tiles. If you wanted a static image, you’d render it into a PNG or JPEG. But Cloud-Optimized GeoTIFF means that GeoTIFFs can be a bit more accessible than they used to be.
Cloud Optimized GeoTIFF (COG)¶
A Cloud Optimized GeoTIFF (COG) is a regular GeoTIFF file, aimed at being hosted on a HTTP file server, with an internal organization that enables more efficient workflows on the cloud. In other words, we can stream the content of a GeoTIFF file from a server into a client application without having to download the full file.
COGs have three major features that are baked into the file:
Tiling¶
Tiles are small, regular, and independent parts of a larger map. They are typically 256x256 pixels in size. COGs have tiles explicitly specified in their format structure, whereas regular GeoTIFFs may not. Tiling is a way to speed up map display because only the tiles that are visible in the current view need to be loaded.
Overviews¶
Overviews are downsampled thumbnail images of the tile. A COG will have many overviews matched to each Zoom Level.
HTTP(s) GET Range Request¶
The HTTP GET Range Request, also known as Byte Serving, allows a client to request specific chunks of the COG using a combination of the tiles and overviews. If you are zoomed into a specific portion of the COG, then you only request the tiles that are visible in the current view. This is the same technology that enables streaming of other media types like video and audio.
Check out the COG Specification
Example COGs on the internet¶
There are numerous cloud based data stores hosting COGs, take a look through a few of these:
-
Microsoft Planetary Computer Catalog - most of the imagery datasets in Planetary Computer are hosted as COGs. You can also view the catalog through the Planetary Computer STAC API browser
-
Planet is streaming out their imagery as COGs. You can also do simple analysis of COGs directly within the Planet platform.
-
This example COG is a drone orthomosaic that is hosted in Cyverse Data Store. We can stream it to the CogEO Viewer.
-
Here is another example of drone imagery COG from OpenAerialMap. The data is hosted on AWS and streamed to the CogEO Viewer.
Applications that can use COGs¶
COGs are geotiffs, so any software application that can read and work with geotiffs will be able to read and work with COGs. This includes QGIS, ArcGIS, and Google Earth Engine.
-
Stream COGs into QGIS see Hands On Lesson
-
You can stream COGs into ArcGIS Pro following this tutorial.
Creating Your Own COGs¶
Creating COGs can be accomplished using:
-
GDAL command line tool. See Hands On Lesson
-
Cogger is a rapid COG generator from GeoTIFF
-
In the python ecosystem rio-cogeo is a RasterIO plugin to create and validate COGs
If I have COGs, do I still need a tile server?¶
The breakthrough of COGs is that they have internal tiling which allows them to be streamed into applications without the need for an additional tile server. For most individuals and small organizations, this should be all you need.
However, there are a few reasons why you might still want to use a tile server with COGs:
-
Performance: Tile servers can cache tiles in memory, which can improve performance by reducing the number of times that tiles need to be read from disk.
-
Scalability: Tile servers can be scaled to handle large numbers of requests.
-
Security: Tile servers can be used to encrypt tiles, which can help to protect them from unauthorized access. If you are streaming COGs to a large number of clients or if you need to ensure that your tiles are secure, then I recommend using a tile server. However, if you are only streaming COGs to a small number of clients and you do not need to worry about security, then you can use the built-in tiles in COGs.
How are COGs different from XYZ and WMTS tiles?¶
Web mapping tile services (WMTS) and XYZ tiles are primarily designed for efficient map display in web environments. Their main goal is to provide quick and seamless map visualizations over the internet by serving small, pre-defined tiles at multiple zoom levels. These tiles are ideal for web maps where users might pan and zoom around the globe, as the small tiles can be fetched and displayed rapidly.
However, for analysis purposes – where users might want to compute statistics, apply algorithms, or extract detailed information from imagery or raster data – these tiling methods are not optimal. The reason is that analysis often requires access to raw, high-resolution data rather than the downsampled or potentially lossy representations provided by these tiles.
That's where formats like Cloud Optimized GeoTIFFs (COGs) come into play. COGs are designed to allow for efficient access to high-resolution raster datasets, making them more suited for analytical purposes. With COGs, one can access and process only specific portions of a large raster without downloading the entire file, making it efficient for cloud-based analysis workflows.
In summary, COGs are designed for efficient access to high-resolution raster data, while XYZ and WMTS tiles are designed for efficient map display.
Hands On¶
Stream COGs into QGIS¶
If you are unfamiliar with QGIS, here are 3 ways to use the open-source program
Install on your local machine
Download and install QGIS on your local machine
Use docker to run QGIS locally without installation
Educational material on software containers and Docker are here.
Installing Docker on your personal computer
We are going to be using virtual machines on the cloud for this course, and we will explain why this is a good thing, but there may be a time when you want to run Docker on your own computer.
Installing Docker takes a little time but it is reasonably straight forward and it is a one-time setup.
Installation instructions from Docker Official Docs for common OS and chip architectures:
Once docker has been installed on your local machine it is easiest to use it on the Command Line Interace (CLI). Run the following commands in your terminal:
this will allow the container access to the X server for display purposes
xhost +
this will run the QGIS container and open the application
docker run --rm -it --name qgis \
-v $(pwd):/data \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-e DISPLAY=unix$DISPLAY \
qgis/qgis:release-3_34 qgis
Run cloud instance of QGIS on Cyverse
Cyverse is a cloud computing and storage tool housed at the University of Arizona. You can sign up for a user account here.
After getting the account, you need to request access to the Visual Interactive Computing Environment (VICE) app. This is a cloud-based desktop environment that allows you to run applications like QGIS in the cloud.
To request VICE access, visit the User Portal look for
and select the REQUEST ACCESS link.
Instructions for launching QGIS and other interactive apps within Cyverse are found here.
Now that you have QGIS running, let's stream a COG into the application.
-
Open QGIS
-
In the "Layers" then "Add Layer" and then "Add Raster Layer"
-
Choose the Source Type and select "Protocol: HTTP(s), cloud, etc" for a file on your computer
-
Enter a valid
https://
in theURl
field for a COG you found online
Here are two COG examples to try. They are both drone orthomosaics that are located in the Cyverse Data Store:
https://data.cyverse.org/dav-anon/iplant/home/jgillan/USGA/imagery_products/hole_17_ortho_cog.tif
https://data.cyverse.org/dav-anon/iplant/home/jgillan/STAC_drone/22_2_ortho_cog.tif
Click "Add" and the COG will be streamed into QGIS
An orthomosaic of a golf course should appear in QGIS
Create COGs with GDAL¶
Geospatial Data Abstration Library (GDAL) is an open source library for working with geospatial data. Most GIS software use GDAL. For regular users, GDAL is a command line tool.
The lastest versions of GDAL (>v3.1) have COG generator installed by default.
Install GDAL
GDAL installation can at times be difficult. When different older python environments are installed on a desktop or laptop GDAL can become broken or incompatiblity issues can come up when installing it.
USGS Windows GDAL Installation Guide
QGIS installs GDAL by default
Anaconda and its package management conda
Docker osgeo/gdal
images are maintained on the Docker Hub
1 Download the example geotif file from the Cyverse Data Store. The file is a 4-band multispectral satellite image from Planet. It was captured on Sept. 6, 2020 over the city of Tucson, AZ and the Santa Catalina Mountains. The file is 1.59 GB in size.
2 Bring it into QGIS to view it.
3 Use gdal to read details of the file
If you managed to install gdal on your local machine, you can run the following command in your terminal:
gdalinfo planet_6sept2020.tif
If you are using docker, you can run the following command in your terminal:
docker run --rm -v /Users:/Users ghcr.io/osgeo/gdal:alpine-small-3.8.0 gdalinfo $PWD/planet_6sept2020.tif
4 Use gdal to create a COG from the sample file
We will be using the gdal_translate command to create COGS. The gdal_translate
command is used to convert raster data between different formats. Here is additional documentation on COG creation options (-co).
for local install:
gdal_translate -of COG -co COMPRESS=LZW planet_6sept2020.tif planet_6sept2020_COG.tif
If using docker:
docker run --rm -v /Users:/Users ghcr.io/osgeo/gdal:alpine-small-3.8.0 gdal_translate -of COG -co COMPRESS=LZW $PWD/planet_6sept2020.tif $PWD/planet_6sept2020_COG.tif
5 Inspect the COG
The new file (2.19GB) should be larger than the original file (1.59GB). This is because the COG has internal tiling and overviews.
Let's look at the details of the new file
gdalinfo planet_6sept2020_COG.tif
or
docker run --rm -v /Users:/Users ghcr.io/osgeo/gdal:alpine-small-3.8.0 gdalinfo $PWD/planet_6sept2020_COG.tif
Buried in all the metadata are a few lines that tell us that the file is a COG. The layout is stated as 'COG' and 'overviews' are listed for each band.
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=PIXEL
LAYOUT=COG
Band 1 Block=512x512 Type=UInt16, ColorInterp=Gray
Min=0.000 Max=8006.000
Minimum=0.000, Maximum=8006.000, Mean=712.188, StdDev=549.828
NoData Value=65536
Overviews: 9414x7918, 4707x3959, 2353x1979, 1176x989, 588x494, 294x247
Use Cyverse to Store and Share Your COG¶
Graphic User Interface (GUI)¶
-
Go to https://de.cyverse.org/
-
Login to Cyverse and then go to the Data Store
- Create a new folder
- Go into the new folder
- Drag and drop the COG into the folder
- Navigate one folder up
- Click the 3dot menu to the far right of your new folder and select "Share"
- Type 'anonymous' in the search bar and select the 'anonymous' user. This will make your COG publically available and able to stream out.
- Once you COG has been uploaded, please click on the 3dot menu to the far right of your COG and select "Public Link"
-
Copy the link and paste it into the CogEO Viewer to view your COG
-
Alternatively, you can paste the link into QGIS to stream the COG into the application
Additional Resources¶
COGS in Production blog post by Sean Rennie