Harmonizome is a multi-omics data integration platform developed by the Ma’ayan Lab. Genomics, proteomics, transcriptomics, metabolomics, and related fields continue to advance technologically and create large quantities of data. Additionally curation efforts have succeeded in mining and aggregating text from decades of biomedical literature into online databases.
It has become more important than ever to harmonize and standardize disparate formats and serve that information to facilitate data reusability. The Harmonizome website allows users to view, download, and visualize data from 138 datasets from some of the most-used omics platforms from across the web. In order to accommodate different types of datasets with a large variety of attribute types, the Harmonizome web application and associated API provide an abstracted framework to store and serve metadata and functional associations from processed datasets.
A key component of the Harmonizome website are gene pages. Gene pages are divided into two sections: a metadata section and a functional association section.
On each gene’s page information about that gene is displayed in the metadata section. Gene symbols have been converted to the symbol used in the NCBI Gene resource. The HGNC family field denotes the family that the gene belongs to from the HUGO Gene Nomenclature Committee, if available. The name field contains the full name of the gene. The description field provides a description of the gene’s function, level of study, functions, and more information if available from NCBI Gene. The synonym field lists any known gene symbols that refer to the same gene. The protein field lists any proteins encoded by the gene, if available, and links to their corresponding Harmonizome page. Finally, the NCBI Gene ID is listed, which links to the NCBI Gene page for the gene. Not all fields will be available for every gene, and any fields that would be empty are not displayed. Links to access the gene’s information and download the gene’s associations through the API are also provided. Lastly, external links to see predicted function, co-expressed genes, and tissue/cell line expression on ARCHS4 are available.
The second part of each gene’s page lists the gene’s functional associations. An expandable menu for each dataset with a gene set functionally associated with the gene is displayed. Each row will contain the name of the dataset and a link to that dataset’s page and a summary of the functional associations listed. Inside the expanded menu, the gene sets with associations are displayed, along with the directions and scores of those associations, if available.
Another key component of the Harmonizome platform are the processed datasets that comprise the knowledge base. Dataset pages are divided into four sections: metadata, data access, visualizations, and gene sets.
The metadata section of the dataset page displays information about the description, measurement, association, category, resource, citation(s), last updated, and statistics for the dataset. The resource field links to a resource page with information about the resource the dataset was sourced from. The citation links will display the corresponding publication in PubMed.
The data access section of the dataset page includes links to API access for the dataset, access to the harmonizomedownloader.py module to download the dataset’s files from Python. The downloads section contains a list of links to download individual files for the dataset. The question mark icon next to each download link can be hovered over to view a description of that download type.
The visualization section displays all available visualizations for the dataset. By default, the visualizations will display in a row of static images. When clicked, the selected visualization will be expanded to provide a better view of the visualization, and where available, provide the interactive version of the visualization. Clicking on another visualization from the display row will replace the currently selected visualization, and clicking on the currently selected visualization in the display row again will minimize it.
Lastly, the gene set section of the dataset page lists all of the gene sets for the dataset. Each of these will have a name which links to the gene set page for that gene set and a description, if available.
Due to the varied nature of data formats and availability between datasets, the processing pipeline is different for each individual dataset. Processing scripts are available as one of the download types, providing a gzip tar directory containing any scripts and mapping files used. The HarmonizomePythonScripts repository on GitHub also houses these processing scripts.
The third key component of Harmonizome are gene set pages. These pages provide details about the information provided for an attribute from a specific dataset. They are accessed through the functional associations found on gene pages and from the gene set sections of dataset pages. Gene set pages are divided into a metadata section and a gene association section.
The metadata section lists the gene set name, the dataset the gene set is from, the dataset category, the attribute type, and a description if available. There is also a link to search for similar terms, which will open a new tab with search results, using the name of the gene set as the query. Finally, another link is provided to access the gene set information through the API.
The gene association section lists all genes associated with the gene set. Depending on the data from the dataset, this can either be a list of associated genes, or the associations can be separated into positive and negative association lists. If the dataset has scores included in the association data, the associated genes will be sorted by their association scores within these tables.
The Harmonizome website is divided into a number of modules to divide and organize the various functionalities available to users.
The Navbar is the main way to navigate between these modules. Each option leads to a different page. The Visualize, Predict, and About options can be clicked to reveal a drop-down with additional options. The Navbar is visible from any Harmonizome page. On some pages, it will also have an embedded search bar with the same functionality as the Search page.
The Search page serves as the homepage for the Harmonizome web application.
The search bar allows you to search for genes, gene sets, and datasets matching an entered term. In the dropdown menu on the left side of the search bar, search results can be filtered to match any of these categories or include all results. The examples can also be clicked to see the results from some commonly searched terms. After a search query is submitted, the user is redirected to a search results page.
The Download page contains a table of all available datasets.
For each dataset, the resource and dataset names, a short description, dataset category, attribute type, and number of views can be displayed. The resource and dataset names are links which will redirect to the corresponding page for that resource or dataset.
The Show Archived Datasets button can be toggled to display older versions of datasets that have been updated. The table is sortable by each column, and the filter bar can be used to search for text in any field.
On this page, select a dataset from a drop-down menu to see a hierarchically clustered heat map with genes as the rows and attributes as the columns. For some datasets, this visualization will be an interactive Clustergrammer embedding, but for most it will be a static image.
On this page, select a dataset from a drop-down menu to see a hierarchically clustered heat map with genes as the rows and columns. The heat map is created using a gene-gene similarity matrix computing the cosine distance between each gene in the dataset.
On this page, select a dataset from a drop-down menu to see a hierarchically clustered heat map with attributes as the rows and columns. The heat map is created using an attribute-attribute similarity matrix computing the cosine distance between each attribute in the dataset.
On this page, select a dataset from a drop-down menu to see an interactive UMAP visualization, with each point representing a gene set in the dataset. TF-IDF gene set vectors were clustered using the Leiden algorithm, and UMAPs are colored by cluster. Hover over each point to see the gene set’s name, cluster, and coordinates.
On this page, select two datasets from drop-down menus to see a hierarchically clustered heat map with attributes from the first dataset as the rows and attributes from the second dataset as the columns.
On this page, select a dataset from a drop-down menu and provide a list of input genes to see a hierarchically clustered heat map with input genes as the rows and attributes from the dataset as the columns. There is a limit on the datasets available and on the number of genes that can be included as an input to limit heat map loading times.
The Predict page houses four machine learning case studies. From the introduction page, view or download the table of results from each of the case studies. Each of the four individual case study pages has information about the classifier model, training data, and results.
The Cross page allows users to compare attributes from different datasets and find unexpected similarities. Two datasets are selected from drop-down menus and a table of pairwise gene set comparisons will be loaded.
Each row in the table represents a crossing of two gene sets. The first column is made up of gene sets from the first selected dataset, the second column lists the size of the first gene set, the third column represents gene sets from the second selected dataset, and the fourth column lists the size of the gene sets from the second dataset. The fifth column represents the value of the Fisher’s Exact Test P-value performed on the genes from each of the two indicated gene sets. The sixth column includes the Jaccard index computed from the genes from each of the two indicated gene sets.The seventh column indicates the number of genes found in both of the two indicated gene sets. The eighth and final column holds the GPT hypothesis generation buttons, which generate a hypothesis using information about each dataset, enriched terms from Enrichr, and the OpenAI API. The table includes up to 5000 rows, representing up to 5000 pairs of gene sets from the two selected datasets sorted by ascending Fisher’s Exact Test P-value.
Clicking the number of overlapping genes will display a modal with the names of the gene sets, a list of the overlapping genes, and buttons to copy the overlapping genes or send the list to Enrichr or Rummagene to perform enrichment analysis.
Clicking the "Generate GPT Hypothesis" button on any row will collect information about the relevant gene sets from metadata in the Harmonizome database, fetch enriched term names and p-values from Enrichr using the overlapping gene list, and prompt OpenAI's GPT-4o model to hypothesize why the overlap between the two gene sets might be significant. When the response is received, it is displayed below the table.
The Chatbot page features an interactive chatbot connected to OpenAI’s GPT-4o LLM model and the Harmonizome database. The chatbot allows users to query Harmonizome with natural language, adding reliability to the LLM’s capabilities. Users can enter chat queries in the bar below the chat history and new messages from the bot and the user will be viewable in the chat window. Chat history is not stored between sessions, but users can reference earlier messages in their queries.
Three example cards are included under the chatbot’s user input bar. Clicking any of the example cards will submit the included text as though the user had submitted it as a query.
While waiting for a response from the chatbot, the submit button will display a loading icon, and the user input bar, submit/loading button, and example cards will become locked until a response is returned and displayed in the message history.
Responses from the chatbot that include names of resources, datasets, and genes from the Harmonizome database are parsed automatically to create links for those instances. These links will be included in the response when displayed in the chat history, and will open the corresponding page in a new tab.
Harmonizome-KG is a new section of the Harmonizome database. The knowledge graph serialization downloads of processed datasets were used to create a knowledge graph UI built on a Neo4j database. Subnetworks can be created by submitting a term (gene or attribute). This will create a new subnetwork with some entities associated with the selected term.
This can be refined by selecting one or more resources to add relations from datasets from that resource. Individual relations can be removed to further customize the subnetwork.
Two terms can also be submitted to create a subnetwork of a path between terms. This is done by specifying a start node, as in the term search,
as well as an end node.
The toolbar provides a number of additional utilies. The limit of edges shown for each relation type can be selected using the slider. Full screen mode can be entered or exited by clicking the icon. Users can switch between network and table view and save either the table or image of the subnetwork. The last section contains options to change the network layout, show edge labels, and a node type legend.
The About page explains the rationale behind the creation and maintenance of Harmonizome, as well as summary statistics about the total associations, resources, datasets, genes, and attributes in the database. There are also interactive visualizations for dataset and attribute categories and the number of gene sets from each resource.
The What’s New page details the updates that have been made to the Harmonizome site. This includes summaries of any new and updated datasets, details and examples of new download types and visualizations, and descriptions of new features.
As an alternative to the website, the Harmonizome knowledge base can also be accessed through a REST API interface. Using this, developers can easily utilize the API to integrate data from Harmonizome into scripts or applications.
The base API URL /api/1.0/ will return a list of supported entities that can be added to the URL string. The list of currently supported entities is:
GET /Harmonizome/api/1.0
{
"version" :1.0,
"entities":[
{
"entity": "attribute",
"href": "/api/1.0/attribute"
},
{
"entity": "dataset",
"href": "/api/1.0/dataset"
},
{
"entity": "gene",
"href": "/api/1.0/gene"
},
{
"entity": "gene set",
"href": "/api/1.0/gene_set"
},
...
]
}
Any of these entities can be appended to the base URL to return a list of that entity type. Each entry in the list will have a name and an href tag.
GET /Harmonizome/api/1.0/gene
{
"count": 56720,
"selection": [0, 100],
"next": "/api/1.0/gene?cursor=100",
"entities": [
{
"symbol": "LOC105377913",
"href": "/api/1.0/gene/LOC105377913"
},
{
"symbol": "LOC105377912",
"href": "/api/1.0/gene/LOC105377912"
},
{
"symbol":"LOC105377911",
"href":"/api/1.0/gene/LOC105377911"
},
...
]
}
To minimize wait times and database queries, entity lists are paginated using a cursor. By default, accessing a list of entities will return the first 100 entities of that type. However, the cursor argument can be added to the URL to specify a start index other than 0.
GET /Harmonizome/api/1.0/gene?cursor=3141
{
"count": 58358,
"Selection": [3141,3241],
"Next": "/api/1.0/gene?cursor=3241",
...
}
If cursor value is larger than the count available for that entity type, an error will be returned.
The href property of an entity returned from an entity list query can be used in the URL to get specific information for that entity. In this case, rather than returning a list of entities, the API response will include a single entry with fields specific to that entity type and information for each.
GET /Harmonizome/api/1.0/gene/nanog
{
"symbol": "NANOG",
"name": "Nanog homeobox",
"ncbiEntrezGeneId": 79923,
"ncbiEntrezGeneUrl": "http://www.ncbi.nlm.nih.gov/gene/79923",
...
}
When accessing information for a gene or gene set, the showAssociations=True argument can be included in the URL to return a list of associated entities in addition to that entity’s base information. All associations have a threhsold value. Unsigned and positive associations have a threshold value of 1, while negative associations have a threshold value of -1. Some datasets also have standardized values available, which are derived from values from the source data. When available, these can serve as a measure of the strength of associations within a dataset or gene set.
GET /Harmonizome/api/1.0/gene/nanog?showAssociations=true
{
"symbol": "NANOG",
...
"associations": [
{
"geneSet": {
"name": "V/Allen Brain Atlas Adult Human Brain Tissue Gene Expression Profiles",
"href": "/api/1.0/gene_set/V/Allen+Brain+Atlas+Adult+Human+Brain+Tissue+Gene+Expression+Profiles"
},
"thresholdValue": 1,
"standardizedValue": 1.33291
},
...
]
}
The harmonizomeapi.py module can be downloaded to interact with the Harmonizome API in Python. Calling get on a supported entity will return information about that entity in the same JSON format as the web interface.The name of the entity can be omitted to return a list of that entity type. To get more of the same entity type, next can be called on the entity list.
from harmonizomeapi import Harmonizome, Entity
pid_dataset = Harmonizome.get(Entity.DATASET, name='PID pathways')
entity_list = Harmonizome.get(Entity.GENE)
more = Harmonizome.next(entity_list)
Harmonizome-KG can also be accessed via the API from https://harmonizome-kg.maayanlab.cloud/api/knowledge_graph which takes the following parameters:
{
"start": "string",
"start_field": "label",
"start_term": "string",
"end": "string",
"end_field": "string",
"end_term": "string",
"limit": 5,
"relation": [
{
"name": "string",
"limit": 5,
}
],
"path_length": 1,
"remove": [
"string"
]
}
The following examples walks through some of the common queries when accessing the API.
Suppose we want to find the immediate neighbors of STAT3. To do this we need to define start and start_term:
import requests
import json
payload = {
"start": "Gene",
# metadata field to query (default: label)
"start_field": "label",
"start_term": "STAT3",
"limit": 10
}
res = requests.get("https://harmonizome-kg.maayanlab.cloud/api/knowledge_graph", params={"filter": json.dumps(payload)})
if res.ok:
results = res.json()
This returns a subnetwork containing the immediate neighbor of STAT3
Suppose we only care about STAT3's participation in GO Biological Process, we can add the relation field:
import requests
import json
payload = {
"start": "Gene",
# metadata field to query (default: label)
"start_field": "label",
"start_term": "STAT3",
"relation": [{
"name": "participates_in_(GO Bio Process 2023)",
"limit": 5
}]
}
res = requests.get("https://harmonizome-kg.maayanlab.cloud/api/knowledge_graph", params={"filter": json.dumps(payload)})
if res.ok:
results = res.json()
Nodes can be removed from the subnetwork by sending their ids to the server.
import requests
import json
payload = {
"start": "Gene",
# metadata field to query (default: label)
"start_field": "label",
"start_term": "STAT3",
"relation": [{
"name": "participates_in_(GO Bio Process 2023)",
"limit": 5
}],
"remove": [
"GO:0045944"
]
}
res = requests.get("https://harmonizome-kg.maayanlab.cloud/api/knowledge_graph", params={"filter": json.dumps(payload)})
if res.ok:
results = res.json()
Suppose we want to find HPO nodes that are connected with the gene STAT3 via shortest path
import requests
import json
payload = {
"start": "Gene",
"start_field": "label",
"start_term": "STAT3",
"end": "HPO",
}
res = requests.get("https://harmonizome-kg.maayanlab.cloud/api/knowledge_graph", params={"filter": json.dumps(payload)})
if res.ok:
results = res.json()
For this example we want to find the shortest path between STAT3 and MAPK1:
import requests
import json
payload = {
"start": "Gene",
"start_field": "label",
"start_term": "STAT3",
"end": "Gene",
"end_field": "label",
"end_term": "MAPK1",
}
res = requests.get("https://harmonizome-kg.maayanlab.cloud/api/knowledge_graph", params={"filter": json.dumps(payload)})
if res.ok:
results = res.json()
Available from any page in the footer, the contact form allows users to contact the maintainers of the site. To submit a question, issue, or feature suggestion, the appropriate option should be selected from the drop-down menu. In the next field, an email to which responses should be directed should be included. Finally, the details box provides space to describe the question, issue, or suggestion. After submitting the request, a confirmation or error screen will appear. Contact submissions are reviewed manually, so please allow a few days for a response.
Harmonizome is available under the Creative Commons BY NC SA 4.0 License for non-commercial use. Users can share and adapt the platform with proper attribution for non-commercial use under the terms of the license. For commercial use, please contact Mount Sinai Innovation Partners.
