We have moved visualisation tools into their own program
to reinforce our commitment to UK spellings, and so that you can rerun visualisations
with different outputs and settings without rerunning the other parts of the code.
Starting with either a full database where you have fitted a model (Fitting new models), or output where you have assigned queries using an existing database (Query assignment), you can create outputs for external interactive visualisation tools:
- Microreact – a genomic epidemiology visualisation tool, displaying clusters, phylogeny and accessory clustering.
- GrapeTree – a tool to visualise strains (designed for cgMLST).
- Phandango – visualisation linking genomes and phylogenies.
- Cytoscape – a network visualisation tool, which can be used to create, view and manipulate a layout of the graph.
At least one of these output types must be specified as a flag.
If you run a visualisation on output from query assignment (Query assignment)
this will not contain all the necessary distances, and they will be calculated before
the visualisation files are produced.
You will see a message
Note: Distance will be extended to full all-vs-all distances.
If you are running multiple visualisations this calculation will be completed every time. To avoid
this re-run your assignment with
--update-db, which will add these distances in permanently.
Some typical commands for various input settings (with
--microreact, but this can
be altered to any output type) with a database
Visualisation of a full database:
poppunk_visualise --ref-db example_db --output example_viz --microreact
Visualisation after query assignment:
poppunk_visualise --ref-db example_db --query-db example_query --output example_viz --microreact
Visualisation when sketches and models are in different folders:
poppunk_visualise --ref-db example_db --previous-clustering example_lineages/example_lineages_lineages.csv \ --model-dir example_lineages --output example_viz --microreact
Visualisation with a lineage model, which has been queried (query-query distances must be provided):
poppunk_visualise --distances example_query/example_query.dists --ref-db example_db \ --model-dir example_lineages --query-db example_lineage_query \ --output example_viz --microreact
Notable modifiers include:
--include-files– give a file with a subset of names to be included in the visualisation.
--external-clustering– other cluster names to map to strains (such as MLST, serotype etc), as described in model fitting and query assignment.
--info-csv– similar to the above, but a CSV which is simply (inner-)joined to the output on sample name.
--rapidnj– the location of a rapidnj binary, used to build the core NJ tree. We highly recommend using this for any tree-building (and is included with the conda install). This defaults to
rapidnjon the PATH. Set blank to use dendropy instead (slower, especially for large datasets).
--accessory-only– use the core or accessory fit from an individually refined model (see Using core/accessory only).
--strand-preserved– querying options used if extra distance calculations are needed. To avoid these, rerun your query with
--microreact flag will create the following files:
- _microreact_clusters.csv – the strain or lineage assignments with headers formatted for microreact, plus anything from
- _core_NJ.nwk – a neighbour-joining tree from the core distances.
- _perplexity20.0_accessory_tsne.dot – a 2D embedding of the accessory distances, produced using t-SNE (in this case with Setting the perplexity parameter for t-SNE 20).
Open https://microreact.org/upload in your browser, and drag and drop these three files to create your visualisation. Here is the result of running the visualisation on the Listeria BGMM model:
poppunk_visualise --ref-db listeria --microreact Graph-tools OpenMP parallelisation enabled: with 1 threads PopPUNK: visualise Loading BGMM 2D Gaussian model Writing microreact output Parsed data, now writing to CSV Building phylogeny Running t-SNE Done
This can be viewed at https://microreact.org/project/8PeGg9fCjZADaAGuNJwU9z:
Useful controls include the tree shape, accessed with the control slider in the top right of the phylogeny page, and the metadata labels, accessed with the ‘eye’ on the right of the page. When visualising lineages, changing the ‘Colour by’ is useful to compare results from different ranks.
In t-SNE an embedding of the accessory genome distances is found which represents local structure of the data. Isolates with similar accessory content will visually appear in clusters together.
The perplexity sets a guess about the number of close neighbours each point
has, and is a trade-off between local and global structure. t-SNE is reasonably
robust to changes in the perplexity parameter (set with
creating microreact output with
however we would recommend trying a few values to get
a good embedding for the accessory distances.
With a lower perplexity of 5, the clustering is too loose, and the strain structure cannot clearly be seen (data):
30 is a good default, but you may wish to try other values, particularly with
larger or smaller datasets. You can re-run the t-SNE using the
command, providing the distances from the previous run:
poppunk_tsne --distances strain_db/strain_db.dists --output strain_db \ --perplexity 20 --verbosity 1
--grapetree flag will create:
- _microreact_clusters.csv – the strain or lineage assignments with headers formatted for grapetree, plus anything from
- _core_NJ.nwk – a neighbour-joining tree from the core distances.
Open https://achtman-lab.github.io/GrapeTree/MSTree_holder.html in your browser, and use the ‘Load files’ button once for each of the files to add the tree and strain assignments to GrapeTree. This will display an unrooted tree with your clusters:
One of GrapeTree’s key features is the ability to collapse branches, and condense information into nodes. By going to Tree Layout -> Branch style -> Collapse branches, and setting the long branch to be shortened, one can obtain a view which shows strain prevalence and relationships:
There is also a handy ‘Export to Microreact’ button in GrapeTree, though this will
not include the accessory embedding, so you may wish to add the
and generate the files yourself.
--phandango flag will create:
- _phandango_clusters.csv – the strain or lineage assignments with headers formatted for phandango, plus anything from
- _core_NJ.tree – a neighbour-joining tree from the core distances.
Open https://www.phandango.net in your browser, and use the ‘Load files’ button once for each of the files to add the tree and strain assignments to GrapeTree. This will display the tree with your clusters:
Press ‘s’ to access settings, and ‘p’ to create an .svg file. Phandango is most useful with a genome (.gff file), and either a plot of recombination, accessory genome analysis or GWAS results. See the documentation for more information.
Cytoscape is different from the above modes as it creates a layout and visualisation of the graph used to create strains from distances. This can be useful for more detailed investigation of network scores, particularly in strains which have less than perfect transitivity.
poppunk_visualise --ref-db listeria --cytoscape Graph-tools OpenMP parallelisation enabled: with 1 threads PopPUNK: visualise Loading BGMM 2D Gaussian model Writing cytoscape output Network loaded: 128 samples Parsed data, now writing to CSV Done
Which will create:
- _cytoscape.csv – the strain or lineage assignments with headers formatted for cytoscape, plus anything from
- _cytoscape.graphml – the network in graphml format.
The .graphml file is an XML file which contains definitions of the nodes (samples)
and edges (within-strain distances) connecting them. If you used
when you fitted your model the edges will be annotated with their Euclidean distances
in the ‘weight’ attribute (which you will need to tell cytoscape). These can be added
poppunk_add_weights script if this flag was not used.
Open cytoscape and drag and drop the .graphml file onto the window to import the network. Import -> table -> file to load the CSV. Click ‘Select None’ then add the ‘id’ column as a key, and any required metadata columns (at least the ‘Cluster’ column) as attributes. Make sure ‘Node Table Columns’ is selected as the data type.
The graphml file does not contain a layout for the graph, that is, positions of nodes and edges are not specified for a visualisation. These will be calculated by cytoscape, automatically for small graphs, and with the ‘Layout’ menu for larger graphs. The ‘Prefuse force directed layout’ or ‘yFiles Organic Layout’ work well. Select the ‘weight’ dropdown to use the edge-lengths when drawing the network.
We have found that making graphs with >10k nodes may exceed the memory on a typical
laptop. To view larger graphs, first splitting into subgraphs of each connected component
is very helpful. Older versions of cytoscape allowed you to split the graph into connected
components, but newer versions have removed this feature. This can be done programmatically
graph-tool in python, or
igraph in R.
Click on ‘Style’ and change the node fill colour to be by cluster, the mapping type as discrete, then right click to autogenerate a colour scheme (‘Random’ is usually best). You can also modify the node size and shape here. Here is the Listeria example, using edge weights in the layout:
If you used assign query mode you will also have a column with ‘Query’ or ‘Reference’, which can be used to map to different shapes or colours:
Adding an info CSV, or loading external tables directly into cytoscapes gives further options for investigating individual strains:
In some cases, edges which are between strain links may have been erroneously included in the network. This could be due to poor model fit, or a poor quality sequence. Use Tools -> NetworkAnalyzer -> Analyze Network to compute information for each node and edge. It may help to analyze connected components separately. They can be split under Tools -> NetworkAnalyzer -> Subnetwork Creation.
Here is an example where an errant node is connecting two clusters into one large cluster, which should be split:
The incorrect node in question has a low CluteringCoefficient and high Stress. The EdgeBetweeness of its connections are also high. Sorting the node and edge tables by these columns can find individual problems such as this.