Users can create GraphFrames from vertex and edge DataFrames.
Vertex DataFrame: A vertex DataFrame should contain a special column named “id” which specifies
unique IDs for each vertex in the graph.
Edge DataFrame: An edge DataFrame should contain two special columns: “src” (source vertex ID
of edge) and “dst” (destination vertex ID of edge).
Both DataFrames can have arbitrary other columns. Those columns can represent vertex and edge
A GraphFrame can also be constructed from a single DataFrame containing edge information.
The vertices will be inferred from the sources and destinations of the edges.
The following example demonstrates how to create a GraphFrame from vertex and edge DataFrames.
The GraphFrame constructed above is available in the GraphFrames package:
The GraphFrame constructed above is available in the GraphFrames package:
Basic graph and DataFrame queries
GraphFrames provide several simple graph queries, such as node degree.
Also, since GraphFrames represent graphs as pairs of vertex and edge DataFrames, it is easy to make
powerful queries directly on the vertex and edge DataFrames. Those DataFrames are made available
as vertices and edges fields in the GraphFrame.
Motif finding refers to searching for structural patterns in a graph.
GraphFrame motif finding uses a simple Domain-Specific Language (DSL) for expressing structural
queries. For example, graph.find("(a)-[e]->(b); (b)-[e2]->(a)") will search for pairs of vertices
a,b connected by edges in both directions. It will return a DataFrame of all such
structures in the graph, with columns for each of the named elements (vertices or edges)
in the motif. In this case, the returned columns will be “a, b, e, e2.”
DSL for expressing structural patterns:
The basic unit of a pattern is an edge.
For example, "(a)-[e]->(b)" expresses an edge e from vertex a to vertex b.
Note that vertices are denoted by parentheses (a), while edges are denoted by
square brackets [e].
A pattern is expressed as a union of edges. Edge patterns can be joined with semicolons.
Motif "(a)-[e]->(b); (b)-[e2]->(c)" specifies two edges from a to b to c.
Within a pattern, names can be assigned to vertices and edges. For example,
"(a)-[e]->(b)" has three named elements: vertices a,b and edge e.
These names serve two purposes:
The names can identify common elements among edges. For example,
"(a)-[e]->(b); (b)-[e2]->(c)" specifies that the same vertex b is the destination
of edge e and source of edge e2.
The names are used as column names in the result DataFrame. If a motif contains
named vertex a, then the result DataFrame will contain a column “a” which is a
StructType with sub-fields equivalent to the schema (columns) of
GraphFrame.vertices. Similarly, an edge e in a motif will produce a column “e”
in the result DataFrame with sub-fields equivalent to the schema (columns) of
It is acceptable to omit names for vertices or edges in motifs when not needed.
E.g., "(a)-->(b)" expresses an edge between vertices a,b but does not assign a name
to the edge. There will be no column for the anonymous edge in the result DataFrame.
Similarly, "(a)-[e]->()" indicates an out-edge of vertex a but does not name
the destination vertex.
An edge can be negated to indicate that the edge should not be present in the graph.
E.g., "(a)-->(b); !(b)-->(a)" finds edges from a to b for which there is no
edge from b to a.
More complex queries, such as queries which operate on vertex or edge attributes,
can be expressed by applying filters to the result DataFrame.
Many motif queries are stateless and simple to express, as in the examples above.
The next examples demonstrate more complex queries which carry state along a path in the motif.
These queries can be expressed by combining GraphFrame motif finding with filters on the result,
where the filters use sequence operations to construct a series of DataFrameColumns.
For example, suppose one wishes to identify a chain of 4 vertices with some property defined
by a sequence of functions. That is, among chains of 4 vertices a->b->c->d, identify the subset
of chains matching this complex filter:
Initialize state on path.
Update state based on vertex a.
Update state based on vertex b.
Etc. for c and d.
If final state matches some condition, then the chain is accepted by the filter.
The below code snippets demonstrate this process, where we identify chains of 4 vertices
such that at least 2 of the 3 edges are “friend” relationships.
In this example, the state is the current count of “friend” edges; in general, it could be any
The above example demonstrated a stateful motif for a fixed-length chain. Currently, in order to
search for variable-length motifs, users need to run one query for each possible length.
However, the above query patterns allow users to re-use the same code for each length, with the
only change being to update the sequence of motif elements (“ab”, “bc”, “cd” above).
In GraphX, the subgraph() method takes an edge triplet (edge, src vertex, and dst vertex, plus
attributes) and allows the user to select a subgraph based on triplet and vertex filters.
GraphFrames provide an even more powerful way to select subgraphs based on a combination of
motif finding and DataFrame filters.
Simple subgraph: vertex and edge filters:
The following example shows how to select a subgraph based upon vertex and edge filters.
Complex subgraph: triplet filters:
The following example shows how to select a subgraph based upon triplet filters which
operate on an edge and its src and dst vertices. This example could be extended to go beyond
triplets by using more complex motifs.
GraphFrames provides the same suite of standard graph algorithms as GraphX, plus some new ones.
We provide brief descriptions and code snippets below.
See the API docs for more details.
Some of the algorithms are currently wrappers around GraphX implementations, so they may not be
more scalable than GraphX. More algorithms will be migrated to native GraphFrames implementations
in the future.
Breadth-first search (BFS)
Breadth-first search (BFS) finds the shortest path(s) from one vertex (or a set of vertices)
to another vertex (or a set of vertices). The beginning and end vertices are specified as
Spark DataFrame expressions.
NOTE: With GraphFrames 0.3.0 and later releases, the default Connected Components algorithm
requires setting a Spark checkpoint directory. Users can revert to the old algorithm using
Run static Label Propagation Algorithm for detecting communities in networks.
Each node in the network is initially assigned to its own community. At every superstep, nodes
send their community affiliation to all neighbors and update their state to the mode community
affiliation of incoming messages.
LPA is a standard community detection algorithm for graphs. It is very inexpensive
computationally, although (1) convergence is not guaranteed and (2) one can end up with
trivial solutions (all nodes are identified into a single community).
Since GraphFrames are built around DataFrames, they automatically support saving and loading
to and from the same set of datasources.
Refer to the Spark SQL User Guide on datasources
for more details.
The below example shows how to save and then load a graph.
Message passing via AggregateMessages
Like GraphX, GraphFrames provides primitives for developing graph algorithms.
The two key components are:
aggregateMessages: Send messages between vertices, and aggregate messages for each vertex.
GraphFrames provides a native aggregateMessages method implemented using DataFrame operations.
This may be used analogously to the GraphX API.
joins: Join message aggregates with the original graph.
GraphFrames rely on DataFrame joins, which provide the full functionality of GraphX joins.
The below code snippets show how to use aggregateMessages to compute the sum of the ages
of adjacent users.
We provide utilities for converting between GraphFrame and GraphX graphs.
See the GraphX User Guide
for details on GraphX.
GraphFrame to GraphX
Conversion to GraphX creates a GraphX Graph which has Long vertex IDs and attributes
of type Row.
Vertex and edge attributes are the original rows in vertices and edges, respectively.
Note that vertex (and edge) attributes include vertex IDs (and source, destination IDs)
in order to support non-Long vertex IDs. If the vertex IDs are not convertible to Long values,
then the values are indexed in order to generate corresponding Long vertex IDs (which is an
The column ordering of the returned Graph vertex and edge attributes are specified by
GraphFrame.vertexColumns and GraphFrame.edgeColumns, respectively.
GraphX to GraphFrame
GraphFrame provides two conversions methods. The first takes any GraphX graph and converts
the vertex and edge RDDs into DataFrames using schema inference. Those DataFrames
are then used to create a GraphFrame.
The second conversion method is more complex and is useful for users with existing GraphX code.
Its main purpose is to support workflows of the following form: (1) convert a GraphFrame to GraphX,
(2) run GraphX code to augment the GraphX graph with new vertex or edge attributes, and
(3) merge the new attributes back into the original GraphFrame.
For example, given:
GraphX Graph[String, Int]graph with a String vertex attribute we want to call “category”
and an Int edge attribute we want to call “count”
We can call fromGraphX(originalGraph, graph, Seq("category"), Seq("count")) to produce
a new GraphFrame. The new GraphFrame will be an augmented version of originalGraph,
with new GraphFrame.vertices column “category” and new GraphFrame.edges column “count” added.