Cycle Chacterization of the Aubry Set for Weakly Coupled Hamilton-Jacobi Systems

We study a class of weakly coupled systems of Hamilton-acobi equations using the random frame introduced in a previous paper of Mitake-Siconolfi-Tran-Yamada. We provide a cycle condition characterizing the points of Aubry set. This generalizes a property already known in the scalar case.


Introduction
True to the title, the object of the paper is to provide a dynamical characterization of the Aubry set associated to weakly coupled Hamilton-Jacobi systems posed on the flat torus T N . We consider the one-parameter family a ij u j (x) = α in T N for every i ∈ {1, · · · , m}, with m ≥ 2 and α varying in R. Here u = (u 1 , · · · , u m ) is the unknown function, H 1 , · · · , H m are continuous Hamiltonians, convex and superlinear in the momentum variable, and A = (a ij ) is a coupling matrix.
As primarily pointed out in [2], [10], this kind of systems exhibits properties and phenomena similar to the ones already studied for scalar Eikonal equations, under suitable assumptions on the coupling matrix.
In particular the minimum α for which the system has subsolutions, to be understood in viscosity or equivalently a.e. sense, is the unique value for which it admits viscosity solutions. We call this threshold value critical, and denote it by β in what follows.
The obstruction to the existence of subsolutions below the critical value is not spread indistinctly on the torus, but instead concentrated on the Aubry set, denoted by A. This fact, proved in [3], generalizes what happens in the scalar case. It has relevant consequences on the structure of the critical subsolutions and allow defining a fundamental class of critical solutions, see Definition 2.2. However, so far, no geometrical/dynamical description of A is available, and the aim of our investigation is precisely to mend this gap.
To deepen knowledge of the Aubry set seems important for the understanding of the interplay between equational and dynamical facts in the study of the system, which is at the core of an adapted weak KAM theory. See [5] for a comprehensive treatment of this topic in the scalar case. This will hopefully allow to attack some open problem in the field, the most relevant being the existence of regular subsolutions. Another related application, at least when the Hamiltonians are of Tonelli type, is in the analysis of random evolutions associated to weakly coupled systems, see [4].
According to [3], there is a restriction in the values that a critical (or supercritical) subsolution can assume at any given point. This is a property which genuinely depends on the vectorial structure of the problem and has no counterpart in the scalar case. Due to stability properties of viscosity subsolutions and the convex nature of the problem, these admissible values make up a closed convex set at any point y of the torus. We denote it by F β (y).
The restriction becomes severe on the Aubry set where F β (y) is a one-dimensional set, while we have proved, to complete the picture, that it possesses nonempty interior outside A, see Proposition 2.6. In a nutshell what we are doing in the paper is to provide a dynamical dressing to this striking dichotomy.
To this purpose, we take advantage of the action functional introduced in [9] in relation to the systems. We also make a crucial use of the characterization of admissible values through the action functional computed on random cycles there established, see Theorem 2.8.
The action functional is defined exploiting the underlying random structure given by the Markov chain with −A as transition matrix. Following the approach of [9], we provide a presentation of the random frame based on explicit computations and avoid using advanced probabilistic notions. This makes the text mostly self contained accessible to readers without specific background in probability.
The starting point is the cycle characterization of the Aubry set holding in the scalar case, see [6]. It asserts that a point is in the Aubry set if and only there exists, for some positive, a sequence of cycles based on it, and defined in [0, t] with t > , on which the action functional is infinitesimal. Of course the role of the lower bound is crucial, otherwise the property should be trivially true for any element of the torus.
To generalize it in the context of systems, we need using random cycles defined on intervals with a stopping time, say τ , as right endpoint. We call it τ -cycles, see Appendix C. This makes the adaptation of the -condition quite painful. To perform the task, we use the notion of stopping time strictly greater than , τ , see Definition 3.5, which seems rather natural but that we were not able to find in the literature. We therefore present in Section 3 some related basic results. We, in particular, prove that the exponential of the coupling matrix related to a τ is strictly positive, see Proposition 3.6. This property will be repeatedly used throughout the paper.
We moreover provide a strengthened version of the aforementioned Theorem 2.8, roughly speaking showing that the τ -cycles with τ are enough to characterize admissible values for critical subsolutions, see Theorem 5.1. This result is in turn based on a cycle iteration technique we explain in Section 4.
The main output is presented in two versions, see Theorems 5.3,5.4, with the latter one, somehow more geometrically flavored, exploiting the notion of characteristic vector of a stopping time, see Definition 3.1.
The paper is organized as follows: in section 2 we introduce the system under study and recall some basic preliminary facts. Section 3 is devoted to illustrate some properties of stopping times and the related shift flows. Section 4 is about the cycle iteration technique. In section 5 we give the main results. Finally the two appendices A and B collect basic material on stochastic matrices and spaces of càdlàg paths. In Appendix C we give a broad picture of the random frame we work within.

Assumptions and preliminary results
In this section we fix some notations and write down the problem with the standing assumptions. We also present some basic results on weakly coupled systems we will need in the following.
We deal with a weakly coupled system of Hamilton-Jacobi equations of the form where m ≥ 2, α is a real constant, A is a constant m × m matrix, the so-called coupling matrix, and H 1 , · · · , ..., H m are Hamiltonians. The H i satisfy the following set of assumptions for all i ∈ {1, · · · , m} : The superlinearity condition (H3) allows to define the corresponding Lagrangians through the Fenchel transform, namely The coupling matrix A = (a ij ) satisfies: (A1) a ij ≤ 0 for every i = j; (A2) m j=1 a ij = 0 for any i ∈ {1, · · · , m}; (A3) it is irreducible, i.e for every W {1, 2, ..., m} there exists i ∈ W and j / ∈ W such that a ij < 0.
Roughly speaking (A3) means that the system cannot split into independent subsytems. We remark that the assumptions (A1) and (A2) on the coupling matrix are equivalent to e −At being a stochastic matrix for any t ≥ 0 and due to irreducibility we get e −At is positive for any t > 0, as made precise in Appendix A.
We will consider (sub/super) solutions of the system in the viscosity sense, see [3], [9] for the definition. We recall that, as usual in convex coercive problems, all the subsolutions are Lipschitzcontinuous and the notions of viscosity and a.e. subsolutions are equivalent.
We now define the critical value β as β = inf{α ∈ R | (HJα) admits subsolutions}, and write down the critical system (HJβ) The critical system (HJβ) is the unique one in the family (HJα), α ∈ R, for which there are solutions. By critical (sub/super) solutions, we will mean (sub/super) solutions of (HJβ).
We deduce from the coercivity condition: The family of subsolutions to (HJα) are equi-Lipschitz-continuous for any α ≥ β. The Lipschitz constant depends on H, A and α.
As already recalled, a relevant property of systems is that not all values in R m are admissible for subsolutions to (HJα) at a given point of the torus. This rigidity phenomenon will play a major role in what follows. We define for α ≥ β and x ∈ T N , where 1 is the vector of R m with all the components equal to 1. It is also apparent from the stability properties of subsolutions and the convex character of the Hamiltonians, that F α is closed and convex at any x. We proceed recalling the PDE definition of Aubry set A of the system, see [3], [9], [12].
We recall the following property: On the contrary, if y ∈ A, the admissible set possesses nonempty interior, which is characterized as follows: Proposition 2.6 Given y / ∈ A, the interior of F β (y) is nonempty, and b ∈ R m is an internal point of F β (y) if and only if there is a critical subsolution u locally strict at y with u(y) = b.
Proof. The values b corresponding to critical subsolutions locally strict at y make up a nonempty set in force of Proposition 2.4, it is in addition convex by the convex character of the system. We will denote it by F β (y).
We denote by u the locally strict critical subsolution with u(y) = b, then there exists 0 < < 1 and δ > 0 such that We fix i and assume where η is a modulus of continuity for (x, p) → H i (x, p) in T N × B(0, β + 1) and β is a Lipschitz constant for all critical subsolutions, see Proposition 2.1. We define w : Further, for j = i and for a.e. x ∈ B(y, ), we have where the last inequality is due to the fact that a ki ≤ 0. The previous computations and (2.5) show that w is a critical subsolution locally strict at y, and this property is inherited by for any λ ∈ [0, 1]. We therefore prove (2.2) setting ν 0 = 2 2 . Taking into account that b + λ 1 ∈ F β (y) for any λ ∈ R and that the vectors e i , i = 1, · · · , m, and −1 are affinely independent, we derive from (2.2) and F β (y) being convex, that b is an internal point of F β (y) and consequently that F β (y) is an open set. Finally it is also dense in F β (y) because if v is any critical subsolution and u is in addition locally strict at y then any convex combination of u and v is locally strict and as λ → 0.
The property of being open, convex and dense in F β (y) implies that F β (y) must coincide with the interior of F β (y), as claimed. 2 We proceed introducing the action functional defined in [9] on which our analysis is based, see Appendices B, C for terminology, notation, definitions and basic facts.
Given α ≥ β and an initial point x ∈ T N , the action functional adapted to the system is where a is any probability vector of R m , τ a bounded stopping time and Ξ a control.
Using the action functional, we get the following characterizations of subsolutions to the system and admissible values: for any pair of points x, y in T N , probability vector a ∈ R m , any bounded stopping time τ and Ξ ∈ K(τ, y − x).

Properties of stopping times
Given a stopping time τ , the push-forward of P a through ω(τ ) is a probability measure on indices {1, · · · , m}, which can be identified with an element of the simplex (denoted by S) of probability vectors in R m . Then a → ω(τ )#P a , defines a map from S to S which is, in addition, linear. Hence, thanks to Proposition A.3, it can be represented by a stochastic matrix, which we denote by e −Aτ , acting on the right, i.e.
(3.1) Definition 3.1 We say that a ∈ S is a characteristic vector of τ if it is an eigenvector of e −Aτ corresponding to the eigenvalue 1, namely a = a e −Aτ .
Remark 3.2 According to Proposition A.4, any stopping time possesses a characteristic vector a, and According to the remark above, Theorem 2.7 takes a simpler form if we just consider expectation operators E a with a characteristic vector. This result will play a key role in Lemma 5.5.
Lemma 3.4 Take τ n as in (C.5). Then e −Aτn → e −Aτ as n goes to infinity.
This implies, taking into account (3.1) and yields the assertion. 2 Definition 3.5 Given any positive constant , we say that τ is strongly greater than , written mathematically as τ , to mean that τ − is still a stopping time, or equivalently τ ≥ a.s. and {τ ≤ t} ∈ F t− for any t ≥ .
Proposition 3.6 Let > 0, i ∈ {1, · · · , m}. Then for every τ in D i , there exists a positive constant ρ, solely depending on and on the coupling matrix, such that Proof. We approximate τ by a sequence of simple stopping times τ n with τ n ≥ τ , as indicated in Proposition C.7. For a fixed n, we then have By the assumption on τ , the set F j := {τ ∈ [(j − 1)/2 n , j/2 n )} ∩ D i belongs to F j 2 n − . By applying Lemma C.5, we therefore get we have in addition e −Aτn → e −Aτ by Lemma 3.4, and consequently This set is compact, and contained in the relative interior of S because e −A is positive by Proposition A.7. Since the components of e i e −Aτ make up the i-th row of e −Aτ , we immediately derive the assertion. 2 According to the previous proposition and Proposition A.5, the the characteristic vector of a τ , for some > 0, is unique and positive.
Remark 3.7 Take τ and denote by ρ -the positive constant satisfying (3.5) for any i, j, according to Proposition 3.6. Then, since e −Aτ is a stochastic matrix, we have Remark 3.8 Let τ , ρ be as in the previous remark. If a is the characteristic vector of τ then we get for any i For any stopping time τ , we consider the shift flow φ τ on D defined by : We proceed by establishing some related properties.
Lemma 3.9 Assume that τ n is a sequence of stopping times converging to τ uniformly in D, then φ τn → φ τ as n → +∞ pointwise in D, with respect to the Skorohod convergence, see Appendix B for the definition.
Proof. We fix ω ∈ D, we set We have for any t This yields the asserted convergence because the g n are a sequence of strictly increasing functions uniformly converging to the identity. Proof. If τ is a simple stopping time, say of the form τ = k t k I(E k ), then and the assertion follows being φ t k measurable for any k, I(E k ) measurable. If τ is not simple then, by Proposition C.5, there exists a sequence of simple stopping times τ n converging to τ uniformly in D, this implies that φ τ is measurable as well, as pointwise limit of measurable maps, in force of Lemma 3.9. 2 We now define the probability measure φ τ #P a , for a ∈ S. The following result generalizes Proposition C.4 to shifts given for stopping times. It will be used in Theorem 4.2 and in Lemma 5.2.
Theorem 3.11 Let a be a probability vector, then We need the following preliminary result: Lemma 3.12 Let a, t, E be a vector in S, a positive deterministic time and a set in F t , respectively, then Proof. We first assume E to be a cylinder, namely for some times and indices, notice that the condition E ∈ F t implies t k ≤ t. We fix i ∈ {1, · · · , m} and consider a cylinder C ⊂ D i , namely for some choice of times and indices. We set We have we also have and we consequently get the relation just depending on E and i. If C is any cylinder, we write where the µ i are defined as in (3.6). Taking into account that µ i ≥ 0 for any i and i µ i = 1, b := i µ i e i ∈ S, we derive from (3.7) This in turn implies by Proposition C.1 showing the assertion in the case where E is a cylinder. If instead E is a multi-cylinder, namely E = ∪ j E j with E j mutually disjoint cylinders then by the previous step Finally, for a general E, we know from Proposition C.1 that there is a sequence of multi-cylinders E n with lim n P a (E n E) = 0. (3.9) We deduce in force of (3.9) which in turn implies that φ t #(P a E n ) weakly converges to φ t #(P a E). Since, by the previous step in the proof φ t #(P a E n ) = P a (E n ) P bn for some b n ∈ S we derive from Proposition C.2 and (3.9) This concludes the proof. If τ = k t k I(E k ) is simple then by Lemma 3.12 Given a general stopping time τ , we approximate it by a sequence of simple stopping times τ n , and, exploiting the previous step, we consider b n ∈ S with φ τn #P a = P bn .
We know from Lemma 3.9 that for any ω in the Skorohod sense, and we derive via Dominate Convergence Theorem for any bounded measurable function f : D → R. Using change of variable formula (C.3) we get This concludes the proof. 2

Cycle iteration
It is immediate that we can construct a sequence of (deterministic) cycles going through a given closed curve any number of times. We aim at generalizing this iterative procedure in the random setting we are working with, starting from τ 0 -cycle, for some stopping time τ 0 . In this case the construction is more involved and requires some details.

Proof.
We argue by induction on j. The property is true for j = 0, assume by inductive step that τ j is a simple stopping time, then by Proposition 3.10 τ j+1 is a random variable, as sum and composition of measurable maps, taking nonnegative values. Assume then the sets F l ∩ {ω | φ τ 0 (ω) ∈ E k } l = 1, · · · , m 0 , k = 1, · · · , m j are mutually disjoint and their union is the whole D. Moreover if which shows that τ j+1 is simple. Since τ 0 , τ j are stopping time then F l ∈ F s l and E k ∈ F t k . By which shows that τ j+1 is a stopping time.
Moreover if τ 0 δ in D i then F l ∩ D i ∈ F s l −δ and consequently The main result of the section is A lemma is preliminary.
Proof. Given j ≥ 1, we preliminarily write We proceed arguing by induction on j. The formula in the statement is true for j = 0. We proceed showing that it is true for j + 1 provided it holds for j ≥ 0. We have, taking into account (4.5) τ j+1 (ω) = τ 0 (ω) + τ j (φ τ 0 (ω)) = τ 0 (ω) + τ j−1 (φ τ 0 (ω)) + τ 0 (φ τ j−1 (φ τ 0 (ω))) = τ j (ω) + τ 0 (φ τ j−1 (φ τ 0 (ω))) = τ j (ω) + τ 0 (φ τ j (ω)) as asserted. 2 Proof. (of Theorem 4. 2) The property is true for j = 0, then we argue by induction on j. We exploit the principle that Ξ j is a control if and only the maps ω → Ξ j (ω)(s) from D to R N are F s -measurable for all s. Given s and a Borel set B of R N , we therefore need to show knowing that Ξ 0 , Ξ j are controls, the first by assumption and the latter by inductive step. We set then we have by the very definition of Ξ j+1 We know that F 1 ∈ F s , (4.8) because τ j is a stopping time and Ξ j a control. Assume now τ j to be of the form then E k \ E = E k or E k \ E = ∅ according on whether t k ≤ s or t k > s and so Consequently, if t k ≤ s, Ξ j+1 (s) is represented in E k by the composition of the following maps By the very definition of the σ-algebra F t , ψ −1 3 (B) ⊂ F s−t k , moreover, since Ξ 0 is adapted then ψ −1 2 (F s−t k ) ⊂ F s−t k and finally ψ −1 1 (F s−t k ) ⊂ F s by Proposition B.1. We deduce, taking also into account that E k ∈ F t k ⊂ F s , that if t k ≤ s then and consequently F 2 , being the union of sets in F s , belongs to F s as well. By combining this information with (4.7), (4.8), we prove (4.6) and conclude that Ξ j+1 is a control. To show that Ξ j+1 is a τ j+1 -cycle, we use the very definition of τ j+1 , Ξ j+1 and write for any ω Due to Ξ j being a τ j -cycle, we have I(ω) = 0 a.s. (4.10) We change the variable in J, setting t = s − τ j (ω), and exploit Lemma 4.3 to get Let E be any set in F and a a positive probability vector. We integrate J(ω) over E with respect to P a using (4.11), replace φ τ j (ω)) by ω via change of variable formula, and exploit Theorem 3.11. We obtain Due to Ξ 0 being a τ 0 -cycle and therefore the integral in the right hand-side of (4.12) is vanishing and so E J(ω) dP a = 0.
Since E has been arbitrarily chosen in F and a > 0, we deduce in force of Lemma C.3 This information combined with (4.9), (4.10) shows that Ξ j+1 is a τ j+1 -cycle and conclude the proof. 2

Dynamical characterization of the Aubry set
In this section we give the main results of the paper on the cycle characterization of the Aubry set. As explained in the Introduction, a key step is to establish a strengthened version of Theorem 2.8, which is based on the cycle iteration technique presented in Section 4.
We break the argument in two parts. The first one is presented in a preliminary lemma.
Proof. We denote by I j the expectation in the left hand side of (5.2) and argue by induction on j. Formula (5.2) is true for j = 0, and we assume by inductive step that it holds for some j ≥ 1.
We know from Proposition C.7 that there is a nonincreasing sequence τ n of simple stopping times with τ n → τ 0 uniformly in D.
We define The τ n are simple stopping times, moreover, since τ 0 is vanishing outside D i and 1 n → 0, we get τ n ≥ τ 0 and τ n → τ 0 uniformly in D, It is clear that Ξ belongs to K(τ n , 0), we further have |L ω(s) (y, 0)| ds.
Owing to the boundedness property of the integrand and that τ n → τ 0 , the right hand-side of the above formula becomes infinitesimal, as n goes to infinity. Therefore the strict negative inequality in (5.6) is maintained replacing τ 0 by τ n , for a suitable n.
Hence we can assume, without loss of generality, that τ 0 appearing in (5.6) satisfies the assumptions of Lemma 5.2 for a suitable δ > 0. Let τ j , Ξ j be as in (4.1), (4.2), we define for any j τ j = τ j + and the τ j are apparently stopping times with τ j , and that Ξ j are τ j -cycles. We have The term U j diverges negatively as j → +∞ by Lemma 5.2, while V j stays bounded, which implies for j large. Taking into account that τ j and Theorem 2.8, the last inequality shows that stopping times strongly greater than and corresponding cycles based at y are sufficient to characterize values b ∈ F β (y). This concludes the proof. 2 Next we state and prove the first main theorem.
where the infimum is taken with respect to any bounded stopping times τ and τ -cycles Ξ. The following properties are equivalent: (i) y ∈ A (ii) the infimum in (5.7) is zero for any index i, any b ∈ F β (y) (iii) the infimum in (5.7) is zero for some i, any b ∈ F β (y).
The assumption that the stopping times involved in the infimum are strongly greater than a positive constant is essentially used for proving (iii) ⇒ (i), while in the implication (i) ⇒ (ii) it is exploited the characterization of admissible values provided in Theorem 5.1.

Proof.
We start proving the implication (i) ⇒ (ii). Let y ∈ A, assume to the contrary that We deduce from Theorem 2.8 that for such i, b. We claim that b + ν e i ∈ F β (y) for any positive ν less than the infimum in (5.8) denoted by η. Taking into account (5.8) and e −Aτ being stochastic, we have for any stopping time τ and τ -cycle Ξ We further get for j = i in force of Theorem 2.8, Combining the information from the above computations with Theorem 5.1, we get the claim, reaching a contradiction with y being in A via Proposition 2.5.
It is trivial that (ii) implies (iii). We complete the proof showing that (iii) implies (i). Let us assume that (5.7) is vanishing for some i and any b ∈ F β (y).
For any positive constant ν, select δ > 0 satisfying δ − ρ ν < 0, where ρ > 0 is given by Proposition 3.6. Notice that we can invoke Proposition 3.6 because we are working with stopping times strongly greater than . We fix b ∈ F β (y) and deduce from (5.7) being zero that there exist a bounded stopping time τ , and a τ -cycle Ξ with Taking into account Remark 3.7 and (5.9), we have which proves that b + ν e i / ∈ F β (y), in view of Theorem 2.8. This proves that b, arbitrarily taken in F β (y), is not an internal point, and consequently that the interior of F β (y) must be empty. This in turn implies that y ∈ A in force of Proposition 2.6. 2 Using expectation operators related to characteristic vectors of stopping times, we have a more geometric formulation of the cycle characterization provided in Theorem 5.3, without any reference to admissible values for critical subsolutions. This is our second main result. where the infimum is taken with respect to any bounded stopping times τ , τ -cycles Ξ and a characteristic vector of τ . where both the infima are taken with respect to any bounded stopping times τ , τ -cycles Ξ, and in (5.12) a is a characteristic vector of τ .
Proof. Let us assume that (5.11) vanishes for any i, then for any δ > 0, any i we find a τ i and τ i -cycles Ξ i with We define a new stopping time τ and a τ -cycle Ξ setting Taking a characteristic vector a = (a 1 , · · · , a m ) of τ , and making convex combinations in the previous formula with coefficients a i , we get taking into account Remark 3.2 L ω (y + I(Ξ), −Ξ) + β ds .
Since we know that the infimum in (5.12) is greater that or equal to 0 thanks to Corollary 3.3 with x = y, the above inequality implies that it must be 0, as claimed. Conversely assume that (5.12) is equal to 0, then for any δ > 0 there is a stopping time τ with characteristic vector a, and a τ -cycle Ξ with From Remark 3.8 and the fact that the expectations in the above inequality must be nonnegative because of Theorem 2.8, we deduce where ρ is the constant appearing in Proposition 3.6. This implies that the infima in (5.11) must vanish for any i. In view of application to coupling matrices of system, we recall Proposition A.6 Given a matrix A and t ≥ 0. Assume (A1) and (A2) hold, then e −At is stochastic.
See [9] for the proof.
Proposition B.1 Given nonnegative constants h, t, we have We also consider that space C(0, +∞; T N ) of continuous paths defined in [0, +∞) with the local uniform convergence. We can associate to it a metric making it a Polish space.
We define a map I : where proj indicates the projection from R N onto T N . It is continuous with respect to the aforementioned metrics, see [9].

Appendix C Random setting
The material of this section is taken from [9]. We are going to define a family of probability measures on (D, F), see [9]. We start from a preliminary result. Taking into account that F, F t are generated by cylinders, we get by the Approximation Theorem for Measures, see [7, Theorem 1.65].
Proposition C.1 Let µ be a finite measure on F. For any E ∈ F, there is a sequence E n of multi-cylinders in F with lim n µ(E n E) = 0, where stands for the symmetric difference. As a consequence we see that two finite measures on D coinciding on the family of cylinders, are actually equal.
Given a probability vector a in R m , namely with nonnegative components summing to 1, we define for any cylinder C(t 1 , · · · , t k ; j 1 , · · · , j k ) a nonnegative function µ a µ a (C(t 1 , · · · , t k ; j 1 , · · · , j k )) = a e −At 1 j 1 k l=2 e −(t l −t l−1 )A j l−1 j l . (C.1) We then exploit that e −At is stochastic to uniquely extend µ a , through Daniell-Kolmogorov Theorem, to a probability measure P a on (D, F), see for instance [13,Theorem 1.2]. Hence, in view of (C.1), we have Proposition C.2 The map a → P a is injective, linear and continuous from S ⊂ R m to the space of probability measures on D endowed with the weak convergence. Consequently, the measures P a are spanned by P i := P e i , for i ∈ {1, · · · , m}, and By (C.1) we also get that P i are supported in D i := C(0; i).
We denote by E a the expectation operators relative to P a , and we put E i instead of E e i . We say that some property holds almost surely, a.s. for short, if it is valid up to P a −null set for some a > 0. We state for later use: Lemma C.3 Let f , a be a real random variable and a positive probability vector, respectively. If E f dP a = 0 for any E ∈ F then f = 0 a.s.
We consider the push-forward of the probability measure P a , for any a ∈ S, through the flow φ h on D. In view of (C.1), one gets: Proposition C.4 For any a ∈ S, h ≥ 0, φ h #P a = P a e −hA .
Accordingly, for any measurable function f : D → R, we have by the change of variable formula The push-forward of P a through ω(t), which is a random variable for any t, is a probability measure on indices. More precisely, we have by (C.1) ω(t)#P a (i) = P a ({ω | ω(t) = i}) = a e −At i for any index i ∈ {1, · · · , m}, so that ω(t)#P a = a e −At .
(C.4) Moreover for b = (b 1 , · · · , b m ) ∈ R m , we have Formula (C.4) can be partially recovered for measures of the type P a E which means P a is restricted to E, where E is any set in F. Lemma C.5 ([9] Lemma 3.4) For a given a ∈ S, E ∈ F t for some t ≥ 0, we have ω(s)#(P a E) = ω(t)#(P a E) e −A(s−t) for any s ≥ t.
Admissible controls: We call control any random variable Ξ taking values in D(0, +∞; R N ) such that (i) it is locally bounded (in time), i.e. for any t > 0 there is M > 0 with sup [0,t] |Ξ(t)| < M a.s.
(ii) it is nonanticipating, i.e. for any t > 0 The second condition is equivalent to require Ξ to be adapted to the filtration F t which means that the map ω → Ξ(ω)(t) from D to R N is measurable with respect F t and the Borel σ-algebra on R N .
We will denote by K the class of admissible controls.
Stopping times: Definition C.6 A stopping time, adapted to F t , is a nonnegative random variable τ satisfying {τ ≤ t} ∈ F t for any t, which also implies {τ < t}, {τ = t} ∈ F t .
We will repeatedly use the following non increasing approximation of a bounded random variable τ by simple stopping times. We set τ n = j j 2 n I({τ ∈ [(j − 1)/2 n , j/2 n )}), (C.5) where I(·) stands for the indicator function of the set at the argument. We have for any j, n {τ n = j/2 n } = {τ < j/2 n } ∩ {τ ≥ (j − 1)/2 n } ∈ F j/2 n , moreover the sum in (C.5) is finite, being τ bounded. Hence τ n are simple stopping times and letting n go to infinity we get: Proposition C.7 Given a bounded stopping time τ , the τ n , defined as in (C.5), make up a sequence of simple stopping times with τ n ≥ τ, τ n → τ uniformly in D.
Given a bounded stopping time τ and a pair x, y of elements of T N , we set K(τ, y − x) = {Ξ ∈ K | I(Ξ)(τ ) = y − x a.s.} , where the symbol − refers to the structure of additive group on T N induced by the projection of R N onto T N = R N /Z N . The controls belonging to K(τ, 0) are called τ -cycles.