0 Introduction

Materials science which started by materials discovered in nature, is quickly evolving to designed materials that are able to adapt to their environment. Starting from atoms and molecules, materials can now be designed from scratch^[1].

The evolution of materials science is driven by the synergy of physics, mechanics, and engineering where image processing reveals important contribution in this challenge^[2]. In particular, it is crucial to design and analyze materials by synthesizing new models from two or three-dimensional image models, with applications in the characterization of the physical assembly process of materials, virtual material design, studying imaging techniques interaction with materials, separating materials appearance from their intrinsic properties, and obtaining a realistic description of the material structure. However, the appearance of the material normally varies with the image scale, image settings, and imaging techniques^[3-5].

Due to the fact that the textured images are diverse^[6], various synthesis approaches have been proposed. They can be divided into three major families: procedural, model-based, and exemplar-based synthesis methods.

The procedural texture synthesis algorithms generate textures using mathematical functions executed with a constant computational cost. Therefore, they are decent for generating the texture of objects in virtual environments, such as video games^[7]. These methods rely on transforming certain pre-defined signals into a desired texture. They are usually used to synthesize structured or unstructured textures, such as the Worley noise^[8].Other noise functions have been also proposed, such as the Gabor noise^[9-10].

The model-based algorithms generate a probabilistic model that can be used to describe and synthesize the texture. The model parameters should be able to capture the essential visual characteristics of the texture. Different models have been proposed in Refs.[11-17].

The exemplar-based algorithms take one or more example textures as input. Most of them perform the synthesis by directly copying pixels or patches from the input images. Thus, these methods produce a new texture which is as similar as possible to the input texture. They are able to synthesize various types of textures, and they can be divided into three families: parametric texture synthesis by analysis^[18-20], pixel by pixel texture synthesis^[21-23], and patch-based texture synthesis^[24-25].

The proposed algorithm in this paper is based on the pixel by pixel texture synthesis method of Wei and Levoy^[21]. The latter relies on a Markov random field, modeling the texture as a stationary and local random process. More precisely, every pixel is predictable from the few pixels in its neighborhood and the spatial statistics are invariant by translation.

This aims to generate an output texture that is locally similar to a sample texture patch. Thus, each synthesized pixel is determined so that local resemblance is preserved between the input and output textures. The algorithm starts from a sample texture and an output image initialized by a white random noise. The synthesis process consists in modifying the noisy output image to make it look like the input sample^[26-27]. To do so, for each output pixel, the neighborhood is extracted and the most similar one is searched for in the exemplar. The corresponding pixel is then copied to the target position in the output texture^[21] (as shown in Fig. 1). The Euclidean distance is adopted to compute the resemblance between two neighborhoods. The synthesis is usually performed using an L-shaped causal neighborhood with a lexicographical scan type (i.e., scan-line). Therefore, the synthesis of a new pixel depends only on the previously synthesized pixels. Note that the neighborhood may contain noise pixels for the first few rows and columns of the output texture. However, as the algorithm progresses, all the other neighborhoods completely contain synthesized pixels. That is, the noise is only used to synthesize the first rows and columns of the output texture, which makes the synthesis result random, and it is then ignored. A non-causal neighborhood with a completely random walk can also be used. It allows the synthesized pixel to free itself from its past, which may consequently result in various configurations.

This study proposes an image synthesis algorithm capable of generating material textures of arbitrary shapes. This algorithm is divided to three stages: 1) A field of second-moment matrices, referred to as the constraint, is created from an arbitrary chosen textured image. 2) A dictionary of intensities and second-moment matrices is constructed from the initial textured image. 3) The generated constraint is used to constrain the synthesis of the textured image by searching the best resemblance within the dictionary.

The remainder of this paper is organized as follows. In Section 1, the second-moment matrix field is reviewed and the creation of the constraint field of second-moment matrices from an imposed arbitrary texture is presented. Section 2 details the construction of the searching dictionary and the synthesis process constrained by the field of second-moment matrices. The obtained results are discussed in Section 3. Finally, the conclusions are drawn in Section 4.

1 Construction of the Constraint Field 1.1 Second-Moment Matrix Field

The second-moment matrix is a mathematical tool commonly used to describe local patterns in several image processing applications such as image synthesis and inpainting^[14]. The second-moment matrix field (S_SMF) of an image I_IM assigns a matrix for each pixel (x, y) of the image. It is defined as the field of local covariance matrices of the first partial derivatives of I_IM expressed as Ref. [15]:

$ \begin{aligned} \boldsymbol{S}_{\mathrm{SMF}}=\boldsymbol{G}_{\boldsymbol{\sigma}} & * \nabla\left(\boldsymbol{I}_{\mathrm{IM}}\right) \nabla\left(\boldsymbol{I}_{\mathrm{IM}}^{\mathrm{T}}\right)= \\ \boldsymbol{G}_{\boldsymbol{\sigma}} & *\left[\begin{array}{cc} I_{\mathrm{IM} x}^2 & I_{\mathrm{IM}_x} I_{\mathrm{IM} y} \\ I_{\mathrm{IM} x} I_{\mathrm{IM} y} & I_{\mathrm{IM} y}^2 \end{array}\right] \end{aligned} $

(1)

where ▽I_IM=[I_IMx, I_IMy] are the gradient fields of image I_IM, G_σ is a Gaussian weighting kernel used to smooth the matrix, [ ]^T is the matrix transpose operator, and '*' denotes the convolution operator.

The orientation factor O_r(x, y) of a second-moment matrix S_SMF(x, y), varying between -π/2 and π/2, is calculated from its first eigenvector [e_x, e_y] as:

$ O_{\mathrm{r}}(x, y)=\tan ^{-1}\left(\frac{\boldsymbol{e}_y}{\boldsymbol{e}_x}\right) $

(2)

An orientation field O_r can then be extracted from a second-moment matrix field S_SMF. O_r consists of an orientation image presenting at each position (x, y), the orientation factor O_r(x, y) of the corresponding matrix S_SMF(x, y) at the same position. The palette used to represent the orientation fields is shown in Fig. 2. Note that this palette is used for all the illustrations and results presented in the sequel.

1.2 Calculation of the Constraint Second-Moment Matrix Field

The first step of the proposed synthesis algorithm consists in constructing the constraint second-moment matrix field S_SMFcon from an imposed arbitrary textured image I_IMcon, which is divided into the following three steps:

1) Calculation of the second-moment matrix field S_SMFcon of I_IMcon and its associated orientation field O_rcon.

2) Calculation of the second-moment matrix field S_SMFin of the input material texture sample I_IMin.

3) Modulation of the orientations of S_SMFin by O_rcon, which is performed as follows:

At each position (x, y) of the field S_SMFin, a rotation for matrix S_SMFin(x, y) by the orientation angle O_rcon(x, y) is performed. The latter is the angle at the corresponding position in the orientation field O_rcon. The constraint field is then computed by updating S_SMFcon, at each position (x, y) as follows:

$ \boldsymbol{S}_{\mathrm{SMFcon}}(x, y) \leftarrow \boldsymbol{R}(x, y) . \boldsymbol{S}_{\mathrm{SMFin}}(x, y) . \boldsymbol{R}(x, y)^{\mathrm{T}} $

(3)

where '.' is the matrix product and R(x, y) is the rotation matrix at position (x, y):

$ \boldsymbol{R}(x, y)=\left[\begin{array}{ll} \cos \left\{O_{\text {con }}(x, y)\right\} & \sin \left\{O_{\text {con }}(x, y)\right\} \\ -\sin \left\{O_{\text {con }}(x, y)\right\} & \cos \left\{O_{\text {con }}(x, y)\right\} \end{array}\right] $

(4)

This process is adopted to ensure that the matrices of S_SMFcon are rotated versions of S_SMFin that faithfully represent the structure of I_IMcon.

2 The Synthesis Process

The second phase of the proposed algorithm involves the establishment of a dictionary of intensities and matrices using the material texture sample I_IMin and its field of second-moment matrices S_SMFin as input. The construction of this dictionary is summarized as follows.

The dictionary $D\left(I_{\text {IMin }}\right)=\left\{N_k, k=1, \ldots, K\right\}$ created from $I_{\text {IMin }}$ is a set of $K$ two-dimensional neighborhoods $N_k$, where $K$ is the number of sites of $I_{\text {IMin }}$. Each neighborhood $N_k$ can be expressed as:

$ N_k=N_k^{I_{\text {IMin }}} \cup N_k^{S_{\mathrm{SMFin}}} $

(5)

where N_k^I_IMin is the k^th intensity neighborhood in I_IMin, and N_k^S_SMFin is the corresponding second-moment matrix neighborhood in S_SMFin.

It is important to mention that causal square neighborhoods or causal L-shaped neighborhoods can be used as intensity and second-moment matrix neighborhoods.

Due to the fact that D(I_IMin) does not necessarily contain the different structures of S_SMFcon, a larger dictionary D_M(I_IMin) which includes D(I_IMin) is then constructed. D_M(I_IMin) consists of neighborhoods extracted from images { Rot_m(I_IMin), m=0, ..., M-1}, where Rot_m(I_IMin) is the rotation of I_IMin by an angle φ_m=2πm/M, with the corresponding second-moment matrices.

Fig. 3 illustrates an example of rotated versions created from an initial material sample I_IMin, where Figs. 3(a)-(c) present an initial anisotropic composite material sample, Fig. 3(b) shows its rotated version by an angle of π/6, and Fig. 3(c) shows its rotated version by an angle of π/3, while the orientation images extracted from the corresponding second-moment matrix fields are shown at the right of each image. Those images represent samples of Rot_m(I_IMin) and their matrix fields that are used to construct D_M(I_IMin). Note that parameter φ_m directly affects the quality of the obtained result, as will be shown in the sequel. More precisely, a lower φ_m value increases the number of the structures in the dictionary, which makes it richer in terms of material patterns. However, increasing the size of the dictionary leads to a higher computational load.

After determining the constraint second-moment field S_SMFcon and the dictionary D_M(I_IMin), the output texture I_IMout is first initialized by values chosen randomly from the input material sample I_IMin. It is then updated during the synthesis process as follows: For each position o=(x, y) in I_IMout, the neighborhood N_o=N_o^I_IMout ∪ N_o^S_SMFcon is extracted. More precisely, N_o^I_IMout is the neighborhood of intensities at position o in I_IMout, and N_o^S_SMFcon is the second-moment matrix neighborhood at the corresponding position in S_SMFcon. The best resemblance is then identified in D_M(I_IMin) and the corresponding pixel is copied to the target position o in I_IMout.

To calculate the similarity between two neighborhoods N_k and N_o, the method involves employing the sum of squared errors (S_SSE) for intensity neighborhoods and the sum of second-moment matrices dissimilarities (S_SSMD) defined in Eq. (7) for the second-moment matrix neighborhoods:

$ \begin{aligned} D\left(N_k, N_o\right)= & W \cdot S_{\mathrm{SSE}}\left(N_k^{I_{\mathrm{IMin}}}, N_o^{I_{\mathrm{IMout}}}\right)+ \\ & (1-W) \cdot \boldsymbol{S}_{\mathrm{SSMD}}\left(N_k^{S_{\mathrm{SMFin}}}, N_o^{S_{\mathrm{SMFcon} }}\right) \end{aligned} $

(6)

where N_k^I_IMin and N_k^S_SMFin are respectively the k^th intensity and second-moment matrix neighborhoods in dictionary D_M(I_IMin), N_o^I_IMout and N_o^S_SMFcon are respectively the intensity and second-moment matrix neighborhoods at position o in I_IMout and the corresponding position in S_SMFcon, W is a weighting factor, and the S_SSMD metric is expressed as:

$ \boldsymbol{S}_{\mathrm{SSMD}}\left(N_1^{S M_1}, N_2^{S M_2}\right)=\sum\limits_{i=1}^{N_M} d_{\mathrm{dist}}\left(N_1^{S M_1}(i), N_2^{S M_2}(i)\right) $

(7)

where N_M is the number of matrices in each neighborhood, N^SM(i) is the i^th second-moment matrix in neighborhood N^SM, and d_dist is one of the four metrics of second-moment matrix resemblance proposed in Ref. [26] (the Euclidean distance is used in this study).

The pseudocode of the whole synthesis algorithm is shown as follows:

Input: I_IMin and I_IMcon

S_SMFcon ←ConstraintCalculation(I_IMin & I_IMcon)

D_M(I_IMin)←DictionaryConstruction(I_IMin & S_SMFin)

I_IMout←NoiseInitialization(I_IMin)

For each position o=(x, y) in I_IMout and S_SMFcon

    P_PIXk←argmax{BestResemblance (N_k vs N_o)}

    I_IMout(x, y) ← P_PIXk

End For

Output: I_IMout

In the case of large-scale material samples, the use of large neighborhoods requires a high computational load. Gaussian pyramids of images and second-moment matrix fields^[27-30] are then used to retranscribe the structures at different scales. Therefore, the D_M(I_IMin) dictionary is constructed using multi-scale neighborhoods extracted from intensity and matrix pyramids. The pyramids are built from M rotated textures and their corresponding second-moment matrix fields. Using the same approach, texture and matrix pyramids are built from I_IMout and S_SMFcon, respectively. The synthesis process begins from the top of the pyramid and descends to the lowest level. More precisely, in order to ensure that the added high frequency details are compatible with the already synthesized low frequency structures, the multi-resolution neighborhood at position o=(x, y) of level L contains its neighborhood at the same level as well as the neighborhood of the corresponding position o′=(x/2, y/2) at the previously synthesized level L+1^[24]. Fig. 4 shows an example of Gaussian pyramid of three levels (left) and multi-resolution causal 5×5 neighborhood (right).

3 Experimental Results

This section presents examples of results obtained using the proposed algorithm. In addition, results obtained with different angular step values are utilized to build the searching dictionary, are shown to highlight the influence of this parameter on the quality of the synthesized images. Moreover, the results obtained by the proposed algorithm are compared with those obtained by directly synthesizing the output texture using the field of second-moment matrices of the arbitrarily imposed texture as a constraint for the synthesis process, is presented.

Fig. 5 shows examples of results obtained using the proposed approach. Figs. 5 (a), (b), (c), show from the 1^st to the 3^rd row, the input material sample (I_IMin) and its second-moment matrix field (S_SMFin) represented by its orientation field, the second-moment matrix field of the arbitrarily imposed texture represented by its orientation image O_rcon, and the output texture (I_IMout).

It can be seen that the synthesized texture, cladded by the patterns of the input material, respects the structures of the imposed texture except that the synthetic texture in Fig. 5(a) presents blocky artifacts due to the abrupt orientation variation (green/blue) of O_rcon.

Fig. 6 shows results obtained using three different values for the rotation angle φ_m used to construct the dictionary. The first row shows the input sample (I_IMin) and its second-moment matrix field (S_SMFin) represented by its orientation field. The 2^nd to 4^th row show the textures obtained by dictionaries built with φ_m of 45°, 20°, and 5°, respectively. The monoscale approach is adopted with 11×11 causal neighborhood, lexicographical scan, and two iterations.

It can be seen that the synthetic texture obtained with φ_m of 45° presents an octogonal structure showing warped patterns. This is attributed to the fact that the built dictionary does not comprise the versions of the input sample and its second-moment matrices that contain enough orientations to reproduce the circular shape of the output texture. When φ_m is equal to 20°, the defects of the synthesized texture disapper.

Although the resulting texture has higher quality compared with the one obtained using a φ_m value of 45 °, unwanted patterns which do not exist in the input sample can be clearly observed. On the contrary, the synthesized texture with φ_m of 5° leads to a texture filled by the intensities of the input sample while being faithful to its characteristics. In summary, it is essential to accurately adapt the angular step to the input sample of the synthesis algorithm.

Fig. 7 shows two different results obtained using imposed textures (I_IMcon) having waves (left) and semicircles (right) shapes. The 1^st row of the Fig. 7 shows the input material sample (I_IMin) and its second-moment matrix field (S_SMFin) represented by its orientation field. The 2^nd to 6^th rows respectively show the arbitrarily imposed texture I_IMcon, its field of second-moment matrices represented by its orientation image O_rcon, the output texture (I_IMdir) directly synthesized using the field of second-moment matrices of I_IMcon as a constraint for the synthesis process, the constraint second-moment matrix field (S_SMFcon) constructed using the proposed approach, and the output texture (I_IMout) obtained using S_SMFcon to constrain the synthesis. For the two results, a searching dictionary with φ_m of 5° is used. Gaussian pyramids of two and three scales are used in the first and second result, respectively. All the presented results are obtained after applying three iterations with a causal neighborhood and a lexicographical scan. It can be observed that the proposed method succeeds in constructing a second-moment matrix field that represents the structures (orientations) of the arbitrarily imposed texture. Consequently, the synthesis process constrained by the obtained field of second-moment matrices leads to restructured textures of arbitrary shapes covered by the patterns of the exemplar, while respecting its dynamics and visual aspects. On the contrary, the synthesis process constrained by the second-moment matrix field of the arbitrary textures leads to images having regular patterns that do not exist in the input material sample. This justifies the importance of the constraint second-moment matrix field instauration algorithm.

It is important to mention that in the proposed algorithm, it is assumed that an input material sample having the same size of the arbitrary texture is available. Otherwise, a solution consists in synthesizing from the input sample, which usually has a small size, a textured image of same size as the arbitrarily imposed texture. The resulting texture is then used for the instauration of the constraint second-moment matrix field.

In order to quantitatively evaluate the obtained results, the peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) index are computed between the input sample and the synthesize texture using the approach consisting of directly synthesizing the output texture by adopting the field of second-moment matrices of the arbitrarily imposed texture as a constraint for the synthesis process (referred to as direct synthesis in the sequel), as well as the synthesized texture obtained using the proposed method. The obtained results are presented in Table 1.

4 Conclusions

This paper proposes a texture synthesis algorithm allowing to synthesize material samples of arbitrary shapes. A constraint second-moment matrix field is first built by modulating the orientations of the input material sample by those of an arbitrary imposed texture. A searching dictionary is then constructed from the input material sample and its second-moment matrix field. In the synthesis process, the constraint second-moment matrix field is used to constrain the synthesis of the output texture by searching the best-match in the dictionary. The obtained results show that the proposed method succeeds in synthesizing textured images of specified shapes while preserving the essential local characteristics of the material exemplar. In future work, we aim at extending the proposed approach to the synthesis of arbitrary-shaped 3D textures from 2D material samples, which would be beneficial in the analysis and design of virtual materials.