1 Introduction

As a learnable model parameterized by θ ∈ ℝn, a standard neural ordinary differential equation (NODE) x.=ϕθ(x,t) is particularly adept at representing complex and nonlinear dynamics (Chen et al., 2018; Liufu et al., 2024), where x ∈ ℝd is the state at time t, x.=dx/dt denotes the time derivative of x, and ϕ(x, t) is a vector field with ϕ ∈ (ℝd×ℝ → ℝd) being a function of x and t. Its strength lies in processing time-variant data and adaptively learning from it. This modeling flexibility renders NODEs great potential for the intricate nature of dynamic systems (Hua et al., 2023; Wang et al., 2023; Jin et al., 2024), enabling a more nuanced understanding of complex dynamic systems.

Despite these strengths, the standard NODE encounters expressivity limitations, failing to model functions like NOT operations (Kidger, 2021; Xu et al., 2023). The NOT operation [i.e., (0, 1) → (1, 0)] involves trajectories that necessarily intersect, presenting a challenge for standard NODEs that cannot model intersecting trajectories due to their first-order nature. The NODE with momentum, which can be regarded as a second-order ODE, improves the expressive capability (Sander et al., 2021): x¨=c0x.+c1ϕθ(x,t), where c0 and c1 are constants. Nonetheless, it can only express limited dynamics due to the linear relationship of x. and ϕθ(x, t). Besides, it cannot model interactions between x. and x. The second-order NODE (SONODE) presented in Norcliffe et al. (2020) seeks to address this limitation by modeling the interactions between x. and x. However, SONODE cannot model interactions between higher-order derivatives and x, and the supervised signal used in training is only the ground-truth value of x, which confines its scope and limits its prediction capability.

To surmount these challenges, we propose a higher-order-derivative-supervised NODE (HiDeS NODE) that is able to model interactions between higher-order derivatives and x. This approach not only expands the expressive range of NODEs but also enhances predictive ability through employing the state vector and its higher-order derivatives as supervised signals, surpassing the modeling and predicting performance of existing NODEs.

This paper evaluates the effectiveness of the HiDeS NODE in the realms of multi-robot systems and opinion dynamics, key areas of dynamic systems, both domains that inherently involve complex interactions and communication (Granha et al., 2022). In multi-robot systems, conventional analytic solutions fall short in high-dimensional control tasks (Károly et al., 2021), such as multi-robot grasping and motion control. NODEs, in contrast, offer a promising avenue for modeling and controlling complex dynamic interactions in a continuous, efficient, and adaptable manner in multi-robot systems. Regarding opinion dynamics research, the primary objective is to decipher the underlying mechanisms and influences that catalyze shifts in opinions. Existing methodologies for learning opinion dynamics overlook the critical prior knowledge that opinion dynamics can be described as an ODE formulated as x.=ϕ(x,t). ODEs are particularly well-suited for modeling the fluid nature of opinion dynamics due to their inherent capacity to capture the dynamics of evolving systems. However, contemporary models employed in learning opinion dynamics underutilize this foundational knowledge. This oversight hampers their ability to effectively capture the nuanced and intricate nature of opinion evolution. Furthermore, the complexities inherent in the evolution of opinions present considerable challenges to the application of existing NODEs in both modeling and forecasting the trajectories of opinion dynamics. The HiDeS NODE conquers these aspects, providing a more effective tool for understanding and predicting opinion evolution.

To bridge these gaps, we propose a new NODE, termed HiDeS NODEs, for modeling and predicting tasks in multi-robot control and opinion dynamics. Figure 1 illustrates the framework of the HiDeS NODE, and Table 1 qualitatively demonstrates the HiDeS NODE's superiority compared with existing NODEs.

Figure 1. Framework of this paper. (Top) Evolution from first to third-order NODEs, highlighting their progressively sophisticated ability to model complex trajectories where higher orders allow for more intricate behaviors. (Middle) Predicting precision with and without using higher-order derivatives as supervised signals, showing the latter's superior approximation of ground truth. (Bottom) Practical applications of HiDeS NODEs in multi-robot systems and opinion dynamics.

Table 1. Comparisons among different NODEs.

The contributions of this paper are demonstrated as follows:

• We propose the HiDeS NODE, a novel approach for modeling the intricacies of dynamics. The HiDeS NODE excels in modeling and predicting interactions among higher-order derivatives within dynamic systems. This advancement provides a more accurate and nuanced representation of dynamic systems.

• The HiDeS NODE integrates higher-order derivatives as supervised signals, significantly enhancing the ability to predict dynamical behaviors.

• We examine the versatility and effectiveness of the proposed HiDeS NODE through its application in two distinct yet complex fields: Multi-robot systems and opinion dynamics. In these fields, the model's ability to capture and predict intricate system dynamics is evaluated.

• To our knowledge, this is the first time that the NODE is introduced for opinion dynamics and multi-robot-system control. Application of the proposed HiDeS NODE to these fields unveils new avenues for both the advancement of NODE methodologies and the nuanced modeling of opinion dynamics and multi-robot-system control.

2 Related work

In this section, we briefly review three lines of research that are close to our work: NODEs, multi-robot-system control methods, and opinion dynamics modeling.

2.1 NODEs

The intersection of neural networks and differential equations, especially interpreting residual networks (ResNets) as discretized ODEs, spurs the development of NODEs (Weinan, 2017; Cui et al., 2023; Ruiz-Balet and Zuazua, 2023). NODEs integrate black-box ODE solvers and neural networks to parameterize the hidden state's derivative. This integration substantially advances time-series modeling, offering robust function approximation and handling of irregular data (Chen et al., 2018; Kidger, 2021). However, standard NODEs encounter representational constraints without dimensionality augmentation, constraining their universal approximation capabilities for certain functions (Dupont et al., 2019).

Research pivots toward higher-dimensional NODEs to overcome these limitations. Momentum-enhanced ResNets, representing second-order NODE extensions, exhibit enhanced capability in modeling non-homeomorphic dynamics and demonstrate improved convergence properties (Sander et al., 2021). In parallel, augmented NODEs, by expanding the solution space, facilitate the learning of more complex functions through simpler dynamic flows, thereby sidestepping the limitations of the vector field's general-representation property (Kidger, 2021). Nonetheless, augmented NODEs introduce challenges in interpretability and alter the loss landscape's structure (Norcliffe et al., 2020). A specific iteration of augmented NODEs, termed second-order NODEs (SONODEs) (Norcliffe et al., 2020), captures more intricate behaviors by integrating second-order dynamics, effectively combining the principles of coupled augmented NODEs. Additionally, the advent of heavy ball NODEs (HBNODEs) (Xia et al., 2021) marks a significant advancement. HBNODEs incorporate the classical momentum accelerated gradient descent method and adeptly mitigate the vanishing gradient problem, thereby enhancing the model's capacity in learning long-term dependencies in sequential data (Xia et al., 2021).

2.2 Multi-robot-system control

Multi-robot systems provide significant benefits in tasks that demand the duplication of effort, risk reduction, or adaptability, offering distinct advantages over single-robot systems (Hichri et al., 2022; Kwa et al., 2022). Multi-robot-system control methods can be categorized into deterministic methods with fixed forms and learning-based methods (Pierpaoli et al., 2021). However, deterministic methods lack flexibility and adaptability in dynamic or unpredictable environments (Liu et al., 2023). In order to overcome these defects, learning methods are increasingly applied to multi-robot control problems. Adaptation methods, for instance, are proposed to enhance trajectory prediction efficiency in multi-agent systems (Aydemir et al., 2023). Furthermore, the parameter-adaptive learning methods are improved through iterative parametric learning controllers (Yu and Chen, 2023). Additionally, neural network-based adaptive learning methods are utilized to learn unknown fault functions, ensuring cooperative tracking in distributed multi-robot systems (Khalili et al., 2020). Despite these advancements, existing methods often fall short in naturally and efficiently modeling the dynamics that often can be described as an ODE, a gap that NODEs can potentially fill.

2.3 Opinion dynamics modeling

Opinion dynamics studies how opinions form and evolve over time through interactions with individuals and environments. Researchers propose various mathematical models to understand and predict the dynamics of opinions. These include continuous-time models such as the DeGroot model (Wu et al., 2023), Hegselmann-Krause model, and bounded confidence model (Kolarijani et al., 2021), as well as discrete-time models like the Ising model, Voter model, and Friedkin and Johnsen model (Baumann et al., 2020; Ao and Jia, 2023; Peng et al., 2023). However, these models, with their fixed forms, lack the flexibility to model the evolution of opinions independently.

In response to these limitations, researchers leverage advances in neural networks to utilize their nonlinear relation approximation ability for learning complex opinion dynamics. An early approach introduces a linear influence model that learns edge influence strength from real data (De et al., 2014). Unlike traditional models, this linear model represents a foundational step in opinion dynamics learning methods, but its simplicity fails to capture the complexity of societal opinion dynamics. Furthering this exploration, SLANT (De et al., 2016; Zhu et al., 2020) introduces a linear model of latent opinions driven by stochastic differential equations (SDEs) using historical, fine-grained event data. Subsequently, SLANT+ (Kulkarni et al., 2017) extends this model with a nonlinear generative model and a network-guided recurrent neural network (RNN) architecture. This model underscores the importance of nonlinearity in designing opinion dynamics models. However, the RNN architecture it relies on faces the challenge of the vanishing gradient problem, hindering long-term predictions of opinion flow. Learnable opinion dynamics model (LODM) (Monti et al., 2020) emerges as a learnable generalization of an opinion dynamics model, combining the causal interpretability of traditional agent-based models with data-driven approaches. Additionally,Okawa and Iwata (2022) introduces the sociologically-informed neural network (SINN), a novel hybrid approach that integrates sociological and social psychological theories with data-driven neural networks to model and predict opinion dynamics in social networks. Despite these advances, current models do not fully exploit the prior knowledge of differential equations in opinion evolution, nor do they effectively model higher-order derivatives.

3 Materials and methods

In this section, formal descriptions and analyses of the proposed HiDeS NODE are provided. Table 2 presents the main symbols and notations used throughout this paper to ensure clarity and ease of understanding.

Table 2. Main symbols and notations.

3.1 Formulation of the HiDeS NODE

The HiDeS NODE has two unique features for modeling and predicting opinion evolution. The first is that the HiDeS NODE is a higher-order NODE that is able to model interactions of higher-order derivatives of the opinion variable x. The second is that the HiDeS NODE adopts higher-order derivatives as supervisory signals to predict opinion evolution better. The HiDeS NODE is described as Equation (1):

[x˙(t)x¨(t)x...(t)⋮x(c)(t)]=ϕθ([x(t)x˙(t)x¨(t)⋮x(c−1)(t)],t),    (1)

where x(t) ∈ ℝd is a time-varying vector representing the opinion of d individuals; t∈[t∨,t^] is the time; Vectors x.(t),x¨(t),x...(t), and x(c)(t) correspond to the first, second, third, and c-th order time derivatives of x, respectively; The function ϕ:ℝcd×ℝ → ℝcd is parameterized by a neural network with the parameter θ ∈ ℝn. Note that [x.(t),x¨(t),x...(t),…,x(c)(t)]⊤∈ℝcd is a concatenation of higher-order derivatives, where the superscript ⊤ means a transpose of a vector, and we call a HiDeS NODE with up to c-th order time derivatives in this concatenation as the HiDeS-c NODE. To enhance readability and avoid redundancy, we may omit “(t)” in certain contexts where the time dependency is understood and does not affect the meaning or clarity of the mathematical expressions.

Remark 1. One advantage of a HiDeS NODE is that it is able to model nonlinear interactions between higher-order derivatives and x. In practice, multiple higher-order derivatives and x can interact with each other. For example, there can be terms like x.⊗x¨ in the vector field, where ⊗ is the Hadamard product.

It can be seen that the standard NODE (Chen et al., 2018) is a HiDeS-1 NODE, and if we just focus on the formulation, SONODE (Norcliffe et al., 2020) can be regarded as a HiDeS-2 NODE. In fact, the HiDeS-2 NODE distinguishes itself from SONODE due to its unique training process.

3.2 Training of the HiDeS NODE

Existing variants of the NODE utilize the ground-truth value of x(t) as the label for training. Differently, the HiDeS NODE adopts the entire [x(t),x.(t),x¨(t),…,x(c-1)(t)]⊤ as the label (the model's prediction [x.(t),x¨(t),x...(t),…,x(c)(t)]⊤ is integrated first to get [x(t),x.(t),x¨(t),…,x(c-1)(t)]⊤). This approach is beneficial for predicting the future evolution of x(t). Training the entire [x(t),x.(t),x¨(t),…,x(c-1)(t)]⊤ gives accurate approximations of all these variables. Since the prediction of the next time step for the HiDeS NODE relies on the entire [x(t),x.(t),x¨(t),…,x(c-1)(t)]⊤, the training strategy of the HiDeS NODE leads to a better prediction performance compared to only training with the ground-truth value of x(t). When the model is predicting the next x(t), utilizing only the ground-truth value of x(t) as the label may lead to an inaccurate result because the basic information it relies on is inaccurate. The inclusion of the derivatives ensures that the model is sensitive to not just the position or condition at a given time but also to the trends and patterns of change, which are critical for forecasting. An explanation is illustrated in Figure 1.

3.3 Inexpressible trajectories of the HiDeS NODE

The superior expressive capability of the HiDeS NODE comes from two aspects.

The first is that lower-order NODEs have limitations in modeling trajectories that require the representation of higher-order dynamics. Consider a trajectory that requires an abrupt change in its acceleration (second derivative of x), which is not expressible in a first-order system but can be expressed in a second-order system [x.(t),x¨(t)]⊤=ϕθ([x(t),x.(t)]⊤,t). Similarly, trajectories requiring changes in the third derivative (jerk) are not expressible in a second-order system but can be captured in a third-order system, and so on. As a result, there exist trajectories that can not be expressed by [x.(t),x¨(t),x...(t),…,x(c)(t)]⊤=ϕθ([x(t),x.(t),x¨(t),…,x(c-1)(t)]⊤,t) but can be expressed by [x.(t),x¨(t),x...(t),…,x(c+1)(t)]⊤=ϕθ([x(t),x.(t),x¨(t),…,x(c)(t)]⊤,t).

The second origin of the superior expressive capability of the HiDeS NODE is that it alleviates the restriction that trajectories cannot cross. One major limitation of the standard NODE is that trajectories under different initial conditions cannot intersect, which constrains its expressive capability. In the following, we show how this constraint is able to be eliminated by the HiDeS NODE.

Theorem 1 (Inexpressible trajectories of a HiDeS-c NODE). Assume that the function ϕθ(ω,t):ℝcd×ℝ→ℝcd with t∈[t∨,t^] is Lipschitz continuous w.r.t. ω ∈ ℝcd. Consider a HiDeS-c NODE governed by Equation (2):

[x.(t),x¨(t),x...(t),…,x(c)(t)]⊤      =ϕθ([x(t),x.(t),x¨(t),…,x(c-1)(t)]⊤,t),      (2)

where x(t) ∈ ℝd is the state vector, ϕθ:ℝcd×ℝ→ℝcd is a continuously differentiable function parameterized by neural network parameters θ. For any two initial conditions ω(t∨)=[x(t∨),x.(t∨),…,x(c-1)(t∨)]⊤ and ω~(t∨)=[x~(t∨),x~.(t∨),…,x~(c-1)(t∨)]⊤, trajectories that require ω(t) and ω~(t) to cross over the interval [t∨,t^] are inexpressible by a HiDeS-c NODE.

Proof. Define the extended state vector ω ∈ ℝcd as ω(t)=[x(t),x.(t),x¨(t),…,x(c-1)(t)]⊤. The HiDeS-c NODE can be represented as Equation (3):

ω.=ϕθ(ω(t),t).    (3)

Given that ϕθ is Lipschitz continuous w.r.t. ω, the Picard-Lindelöf theorem (Anil Kumar et al., 2022; Zhang et al., 2023) assures the existence of a unique solution ω(t) for a given initial condition ω(0). This uniqueness implies that for any two distinct initial conditions (ω(t∨)) and ω~(t∨), the resulting trajectories ω(t) and ω~(t) do not cross over [t∨,t^]. As a result, trajectories that require ω(t) and ω~(t) to cross over the interval [t∨,t^] are inexpressible by a HiDeS-c NODE. The proof is thus completed.          ⎕

Remark 2. From Theorem 1, it can be seen that as the order c increases, the degree of freedom for avoiding the crossing of x(t) and x~(t) increases. In practice, there are some trajectories cross in the x-t space or in the phase space [i.e., x-x.-x¨-…-x(c−1)-t space], so the HiDeS-c NODE provides a better capability to model these dynamics compared with the standard NODE (a HiDeS-1 NODE) (Chen et al., 2018) and other second-order NODEs (HiDeS-2 NODEs) (Norcliffe et al., 2020; Sander et al., 2021). An intuitive understanding is that additional dimensions provide new directions to make trajectories elude each other, which is illustrated in Figure 1. Consider two actual evolution trajectories under different initial conditions, x*(t) and x~*(t), which may intersect at some time points if there are no restrictions. However, for c = 1, the trajectories x(t) and x~(t) generated by the standard NODE cannot intersect due to its nature of first-order ODEs. This limitation means they cannot accurately approximate x*(t) and x~*(t) in cases where the actual trajectories intersect. Theorem 1 from our manuscript implies similar limitations for higher-order NODEs, but with increasing order c, the trajectories have more freedom, reducing the limitations.

3.4 The HiDeS NODE's utilization of historical information

Due to the introduction of higher-order derivatives, the HiDeS NODE implicitly uses historical state information for predicting the next state. The reason is that higher-order derivatives can be approximated by historical states. For example, the first derivative x. at the k-th moment tk can be approximated as x.(tk)≈(x(tk)-x(tk-1))/Δt. Similarly, the second derivative x¨ can be approximated as Equation (4):

x¨(tk)≈(x(tk)-x(tk-1)Δt)-(x(tk-1)-x(tk-2)Δt)Δt.    (4)

Iteratively, we have Equation (5):

x(c)(tk)≈∑i=0c(−1)i( ic)x(tk−i)/(Δtc),    (5)

where the binomial coefficient ( ic)=c!/(i!(c-i)!) represents the combinatorial number of ways to choose i elements from a set of c elements. Consequently, ϕθ([x(t),x.(t),x¨(t),…,x(c-1)(t)]⊤,t) in the HiDeS NODE (Equation 1) can be approximated as a function of historical states as in Equation (6):

[x.(t),x..(t)